<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@489ada2da91a4cd48ecb3ff574aea3ac" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">Introduction</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@a1ff940cc7dc45779d0a97c38342bad1">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@a1ff940cc7dc45779d0a97c38342bad1" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Introduction</h3>
<p>Once you’ve finished your primary analysis, your results and design decisions, such as inclusion criteria and cutoff thresholds still need to be validated. It can be easy to produce misleading conclusions if you forgo examining your analysis. In particular, you should answer these questions:</p>
<ul>
<li>Does the data meet the assumptions of your model? </li>
<li>Would you get the same result with slightly different data? </li>
<li>How confident should you be in your results? </li>
<li>How much would your results change if your inclusion criteria were slightly different?</li>
</ul>
<p>These steps are especially important if you have a causal interpretation of your results, since causal inference is often limited by the assumptions made in study design and analysis and this is particularly pronounced when working with observational health data.</p>
<p></p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@a1a3177ff1774e1ba7a001d0b26f44c7">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@a1a3177ff1774e1ba7a001d0b26f44c7" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Learning Objectives</h3>
<ul>
<li>Appreciate that all models possess inherent limitations for generalizability.</li>
<li>Understand the assumptions for making causal inferences from available data.</li>
<li>Check model fit and performance</li>
</ul>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@db0a5ab61cf24927a802c5823a625b36">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@db0a5ab61cf24927a802c5823a625b36" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Credits</h3>
<p>Lawrence Baker (written content and images) & Marie Charpignon (R code)</p>
<p> Justin D. Salciccioli, Yves Crutain, Matthieu Komorowski and Dominic C. Marshall. MIT Critical Data - Secondary Analysis of Electronic Health Records, Chapter 17 (2016).</p>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@a17ae834338c43aeae7587b5f64ce815" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">What are Sensitivity Analysis and Model Validation?</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@ead765da36a54369aa62bdba270396a0">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@ead765da36a54369aa62bdba270396a0" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>What are Sensitivity Analysis and Model Validation?</h3>
<p>Sensitivity Analysis and Model Validation are both attempts to assess the appropriateness of a particular model specification and to appreciate the strength of the conclusions being drawn from such a model.</p>
<p>Model validation is the process of assessing how well your model fits within your specific research dataset. It is checking the appropriateness of your model.</p>
<p>Sensitivity analysis is the process of gaining confidence in the generalizability of the result of the primary analysis and is important in situations where a model is likely to be used in a future research investigation or in clinical practice.</p>
<p>This module will first talk through model validation and will then touch on how to increase the chance that your results would be similar in other datasets, through sensitivity analysis.</p>
<p></p>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@6f24c88eea2e4815be737abeeca8ff73" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">Model Assumptions</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@d27c8e9acf8a4d1f9b820ab120031f92">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@d27c8e9acf8a4d1f9b820ab120031f92" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Model Assumptions</h3>
<p>The first step to verifying model validity is to check whether your data and analysis meet the model's assumptions. The video below shows an example for linear regression, but for any method, you should make a checklist of your assumptions and check each one against your results and dataset.</p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679">
<div class="xblock xblock-public_view xblock-public_view-video xmodule_display xmodule_VideoBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="video" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Video"}
</script>
<h3 class="hd hd-2">Assumptions for Linear Regression</h3>
<div
id="video_604d47c61e0b45c988422e853092d679"
class="video closed"
data-metadata='{"saveStateUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679/handler/xmodule_handler/save_user_state", "lmsRootURL": "https://openlearninglibrary.mit.edu", "publishCompletionUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679/handler/publish_completion", "streams": "1.00:JKRpiA85sts", "duration": 0.0, "recordedYoutubeIsAvailable": true, "transcriptAvailableTranslationsUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679/handler/transcript/available_translations", "captionDataDir": null, "ytApiUrl": "https://www.youtube.com/iframe_api", "speed": null, "end": 0.0, "completionPercentage": 0.95, "autoAdvance": false, "transcriptLanguage": "en", "prioritizeHls": false, "autohideHtml5": false, "ytTestTimeout": 1500, "transcriptLanguages": {"en": "English"}, "savedVideoPosition": 0.0, "sources": [], "completionEnabled": false, "saveStateEnabled": false, "generalSpeed": 1.0, "autoplay": false, "poster": null, "showCaptions": "true", "transcriptTranslationUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679/handler/transcript/translation/__lang__", "ytMetadataEndpoint": "", "start": 0.0}'
data-bumper-metadata='null'
data-autoadvance-enabled="False"
data-poster='null'
tabindex="-1"
>
<div class="focus_grabber first"></div>
<div class="tc-wrapper">
<div class="video-wrapper">
<span tabindex="0" class="spinner" aria-hidden="false" aria-label="Loading video player"></span>
<span tabindex="-1" class="btn-play fa fa-youtube-play fa-2x is-hidden" aria-hidden="true" aria-label="Play video"></span>
<div class="video-player-pre"></div>
<div class="video-player">
<div id="604d47c61e0b45c988422e853092d679"></div>
<h4 class="hd hd-4 video-error is-hidden">No playable video sources found.</h4>
<h4 class="hd hd-4 video-hls-error is-hidden">
Your browser does not support this video format. Try using a different browser.
</h4>
</div>
<div class="video-player-post"></div>
<div class="closed-captions"></div>
<div class="video-controls is-hidden">
<div>
<div class="vcr"><div class="vidtime">0:00 / 0:00</div></div>
<div class="secondary-controls"></div>
</div>
</div>
</div>
</div>
<div class="focus_grabber last"></div>
<h3 class="hd hd-4 downloads-heading sr" id="video-download-transcripts_604d47c61e0b45c988422e853092d679">Downloads and transcripts</h3>
<div class="wrapper-downloads" role="region" aria-labelledby="video-download-transcripts_604d47c61e0b45c988422e853092d679">
<div class="wrapper-download-transcripts">
<h4 class="hd hd-5">Transcripts</h4>
<ul class="list-download-transcripts">
<li class="transcript-option">
<a class="btn btn-link" href="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679/handler/transcript/download" data-value="srt">Download SubRip (.srt) file</a>
</li>
<li class="transcript-option">
<a class="btn btn-link" href="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@604d47c61e0b45c988422e853092d679/handler/transcript/download" data-value="txt">Download Text (.txt) file</a>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@eb775065014b46e6ad84071f737d0a04">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@eb775065014b46e6ad84071f737d0a04" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Speaker Credit</h3>
<p>Speaker: Jesse Raffa - all videos in this section</p>
</div>
</div>
<div class="vert vert-3" data-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@afdfca89d7184bcaae4ef155c054093c">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@afdfca89d7184bcaae4ef155c054093c" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="problem" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="True" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_afdfca89d7184bcaae4ef155c054093c" class="problems-wrapper" role="group"
aria-labelledby="afdfca89d7184bcaae4ef155c054093c-problem-title"
data-problem-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@afdfca89d7184bcaae4ef155c054093c" data-url="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@problem+block@afdfca89d7184bcaae4ef155c054093c/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="1"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="afdfca89d7184bcaae4ef155c054093c-problem-title" aria-describedby="block-v1:MITx+HST.953x+3T2020+type@problem+block@afdfca89d7184bcaae4ef155c054093c-problem-progress" tabindex="-1">
Question 1: Assumptions in Logistic Regression
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.953x+3T2020+type@problem+block@afdfca89d7184bcaae4ef155c054093c-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><p>Now that we have seen an example with linear regression, let's test if you know the assumptions for another commonly used model. Imagine you had completed an analysis using logistic regression to classify records into one of two classes.</p>
<div class="choicegroup capa_inputtype" id="inputtype_afdfca89d7184bcaae4ef155c054093c_2_1">
<fieldset aria-describedby="status_afdfca89d7184bcaae4ef155c054093c_2_1 description_afdfca89d7184bcaae4ef155c054093c_1_1">
<legend id="afdfca89d7184bcaae4ef155c054093c_2_1-legend" class="response-fieldset-legend field-group-hd">The validity of your analysis rests on meeting certain assumptions which make logistic regression an approprtiate model. What are these assumptions?</legend>
<p class="question-description" id="description_afdfca89d7184bcaae4ef155c054093c_1_1">. There may be multiple correct and incorrect answers, check all that apply.</p>
<div class="field">
<input type="checkbox" name="input_afdfca89d7184bcaae4ef155c054093c_2_1[]" id="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_0" class="field-input input-checkbox" value="choice_0"/><label id="afdfca89d7184bcaae4ef155c054093c_2_1-choice_0-label" for="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_0" class="response-label field-label label-inline" aria-describedby="status_afdfca89d7184bcaae4ef155c054093c_2_1 description_afdfca89d7184bcaae4ef155c054093c_1_1"> The data follows a normal distribution
</label>
</div>
<div class="field">
<input type="checkbox" name="input_afdfca89d7184bcaae4ef155c054093c_2_1[]" id="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_1" class="field-input input-checkbox" value="choice_1"/><label id="afdfca89d7184bcaae4ef155c054093c_2_1-choice_1-label" for="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_1" class="response-label field-label label-inline" aria-describedby="status_afdfca89d7184bcaae4ef155c054093c_2_1 description_afdfca89d7184bcaae4ef155c054093c_1_1"> The data is not autocorrelated
</label>
</div>
<div class="field">
<input type="checkbox" name="input_afdfca89d7184bcaae4ef155c054093c_2_1[]" id="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_2" class="field-input input-checkbox" value="choice_2"/><label id="afdfca89d7184bcaae4ef155c054093c_2_1-choice_2-label" for="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_2" class="response-label field-label label-inline" aria-describedby="status_afdfca89d7184bcaae4ef155c054093c_2_1 description_afdfca89d7184bcaae4ef155c054093c_1_1"> The data is not hierarchically clustered
</label>
</div>
<div class="field">
<input type="checkbox" name="input_afdfca89d7184bcaae4ef155c054093c_2_1[]" id="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_3" class="field-input input-checkbox" value="choice_3"/><label id="afdfca89d7184bcaae4ef155c054093c_2_1-choice_3-label" for="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_3" class="response-label field-label label-inline" aria-describedby="status_afdfca89d7184bcaae4ef155c054093c_2_1 description_afdfca89d7184bcaae4ef155c054093c_1_1"> The data is homoscedastic
</label>
</div>
<div class="field">
<input type="checkbox" name="input_afdfca89d7184bcaae4ef155c054093c_2_1[]" id="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_4" class="field-input input-checkbox" value="choice_4"/><label id="afdfca89d7184bcaae4ef155c054093c_2_1-choice_4-label" for="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_4" class="response-label field-label label-inline" aria-describedby="status_afdfca89d7184bcaae4ef155c054093c_2_1 description_afdfca89d7184bcaae4ef155c054093c_1_1"> The data follows a log-normal distribution
</label>
</div>
<div class="field">
<input type="checkbox" name="input_afdfca89d7184bcaae4ef155c054093c_2_1[]" id="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_5" class="field-input input-checkbox" value="choice_5"/><label id="afdfca89d7184bcaae4ef155c054093c_2_1-choice_5-label" for="input_afdfca89d7184bcaae4ef155c054093c_2_1_choice_5" class="response-label field-label label-inline" aria-describedby="status_afdfca89d7184bcaae4ef155c054093c_2_1 description_afdfca89d7184bcaae4ef155c054093c_1_1"> The features in the data are not collinear
</label>
</div>
<span id="answer_afdfca89d7184bcaae4ef155c054093c_2_1"/>
</fieldset>
<div class="indicator-container">
<span class="status unanswered" id="status_afdfca89d7184bcaae4ef155c054093c_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Question 1: Assumptions in Logistic Regression" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_afdfca89d7184bcaae4ef155c054093c" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_afdfca89d7184bcaae4ef155c054093c">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="afdfca89d7184bcaae4ef155c054093c-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="afdfca89d7184bcaae4ef155c054093c-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="afdfca89d7184bcaae4ef155c054093c-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="False">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@8c0fed862e894c6d8174781f6c906c7f" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">Bias and Variance</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@504262fd8ad146649314ca9b735a81ac">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@504262fd8ad146649314ca9b735a81ac" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Bias and Variance</h3>
<p>Even if your model is correctly specified, you still need to know that your results are reproducible and not the product of a particular modelling choice or training sample of data. Part of this is navigating the variance-bias tradeoff:</p>
<p>A model with high bias fails to accurately model the data, this could be because the model is wrongly specified (i.e. your data do not meet the assumptions of your model), or because the features in your dataset are not predictive. A model with high bias is underfit. Variance is related to the predictive ability of your model in new, unseen, data. A model with high variance overfitted and may fail to generalize to new examples. Often decreasing bias will increase variance and vice-versa, you must make a compromise between the two.</p>
<p>Underfitting (bias) can occur when there are too few features and leads to bias - the model’s predictions systemically inaccurate because the underlying relationships have not be captured. If you suspect your are underfitting, try adding new features or specifying a different model. Merely adding more training examples will not fix bias, because this will not change the underlying relationship between the model and the data.</p>
<p>Overfitting can occur when the model has been trained too specifically on the training data, which leads to poor generalisability, also known as high variance. If your model has high variance then try to reduce its complexity, by reducing the number of features or reducing dimensions through methods such as principal component analysis. Adding more training examples will also decrease variance.</p>
<p>Examples of bias and variance are shown in figure 2.10.1.</p>
<h5 style="text-align: center;"><img height="247" width="700" src="/assets/courseware/v1/4c2a1651c3d76a01185c426a6d726e47/asset-v1:MITx+HST.953x+3T2020+type@asset+block/Capture.JPG" alt="Underfitting vs Overfitting" /></h5>
<h4 style="text-align: left;">Figure 2.10.1 Examples of bias and variance in data with a quadratic relationship.</h4>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@0f09844a9ec44b6c93081f6565cd72b4" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">Evaluating Model Performance</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@fd88759567724156bf01de308f1d0ced">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@fd88759567724156bf01de308f1d0ced" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Evaluating Model Performance</h3>
<p>We need to know how to evaluate model accuracy. There are many ways to do this, three of the most simple and commonly used methods are:</p>
<ul>
<li>Accuracy, the proportion of records which were correctly labeled (in a classification problem)</li>
<li>R<sup>2</sup>, the proportion of the variance in the outcome parameter explained by the model (in a regression problem)</li>
<li>The area under the Receiver Operating Characteristic Curve (AU-ROC) (in a classification problem)</li>
</ul>
<p></p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@b31069bd65934f22b802e5ddeed1a9b4">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@b31069bd65934f22b802e5ddeed1a9b4" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Sensitivity and Specificity</h3>
<p>In binary classification problems, which is a problem where we are predicting either a '1' or a '0 ' label, predictions can fall into one of four categories:</p>
<ul>
<li>TN=True negatives (correct 0 classifications)</li>
<li>FN=False negatives (where the model predicted a 0, but in reality the label was a 1)</li>
<li>TP=True positives (correct 1 classifications)</li>
<li>FP=False positives (where the model predicted a 1, but in reality the label was a 0)</li>
</ul>
<p>Accuracy is the proportion of all records which are true negatives or true positives (TN+TP)/(TN+TP+FN+FP).</p>
<p>One issue with accuracy is that it is not a good metric of performance if the dataset is unbalanced. For instance, imagine you were predicting mortality in a hospital with a 1% mortality rate. A model which predicted everyone to be a 0 would be 99% accurate! To protect against this flaw, we should also always look at the specificity and sensitivity of our model.</p>
<p>Sensitivity (also known as the true positive rate) measures the proportion of positives that are correctly identified (TP)/(TP+FN)</p>
<p>Specificity (also known as the true negative rate) measures the proportion of negatives that are correctly identified (TN)/(TN+FP)</p>
<p>A more robust way to investigate accuracy, sensitivity and specificity is though receiver operating characteristic (ROC) curves.</p>
<p></p>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@450c8608e3fb4af49a858869903dc2cc">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@450c8608e3fb4af49a858869903dc2cc" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Receiver Operating Characteristic Curve</h3>
<p>Most classification algorithms, rather than producing a binary prediction (1 or 0) actually produce a probabilistic prediction that a record falls into a class (e.g. 0.7 chance of a 1, 0.3 chance of a 0). There is then a threshold, usually 0.5 by default, which determines whether the prediction is a 1 or a 0. The receiver operating characteristic curve is a plot of the false positive rate against the true positive rate as the threshold is varied. This varying threshold captures the idea that, for any classification model, the true positive rate can always be increased if a corresponding increase in the false positive rate can be tolerated.</p>
<p>A ROC curve is shown in Figure 2.10.2:</p>
<p><img height="313" width="483" src="/assets/courseware/v1/5ec746d7e3a3ba921905ae43840cb2da/asset-v1:MITx+HST.953x+3T2020+type@asset+block/image2.png" alt="An example of a ROC curve" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<h4>Figure 2.10.2: Receiver operating characteristic curve</h4>
<p>The orange curve is a plot of all the potential respective FPRs and TPRs we could achieve by varying the discrimination threshold. In this case we could choose a threshold which gave an FPR of 0.2 and a TPR of approximately 0.6.</p>
<p>The area under this curve gives the AU-ROC score, which varies between 1 (for a perfectly predictive model) and 0.5 (for a model which is no better than random). Scores are meaningful only the context of the problem and relative to the performance of other models, but an AU-ROC > 0.9 would generally be considered very predictive.</p>
<p></p>
</div>
</div>
<div class="vert vert-3" data-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@564da5ccee4f4937ad6f7f16249fd748">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@564da5ccee4f4937ad6f7f16249fd748" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="problem" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="True" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_564da5ccee4f4937ad6f7f16249fd748" class="problems-wrapper" role="group"
aria-labelledby="564da5ccee4f4937ad6f7f16249fd748-problem-title"
data-problem-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@564da5ccee4f4937ad6f7f16249fd748" data-url="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@problem+block@564da5ccee4f4937ad6f7f16249fd748/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="4"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="564da5ccee4f4937ad6f7f16249fd748-problem-title" aria-describedby="block-v1:MITx+HST.953x+3T2020+type@problem+block@564da5ccee4f4937ad6f7f16249fd748-problem-progress" tabindex="-1">
Matching ROC Curves
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.953x+3T2020+type@problem+block@564da5ccee4f4937ad6f7f16249fd748-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><div class="choicegroup capa_inputtype" id="inputtype_564da5ccee4f4937ad6f7f16249fd748_2_1">
<fieldset aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_2_1">
<legend id="564da5ccee4f4937ad6f7f16249fd748_2_1-legend" class="response-fieldset-legend field-group-hd">A series of ROC curves are show below, what best describes the model performance in each case?
<img src="/assets/courseware/v1/0069d7a00e892b42bd30bce932e76cc1/asset-v1:MITx+HST.953x+3T2020+type@asset+block/image5.png" alt="A ROC curve"/></legend>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_2_1" id="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_0" class="field-input input-radio" value="choice_0"/><label id="564da5ccee4f4937ad6f7f16249fd748_2_1-choice_0-label" for="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_0" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_2_1"> A model with low predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_2_1" id="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_1" class="field-input input-radio" value="choice_1"/><label id="564da5ccee4f4937ad6f7f16249fd748_2_1-choice_1-label" for="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_1" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_2_1"> A model with high predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_2_1" id="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_2" class="field-input input-radio" value="choice_2"/><label id="564da5ccee4f4937ad6f7f16249fd748_2_1-choice_2-label" for="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_2" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_2_1"> A model which outputs 1 or 0 at random
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_2_1" id="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_3" class="field-input input-radio" value="choice_3"/><label id="564da5ccee4f4937ad6f7f16249fd748_2_1-choice_3-label" for="input_564da5ccee4f4937ad6f7f16249fd748_2_1_choice_3" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_2_1"> A model which predicts the data perfectly
</label>
</div>
<span id="answer_564da5ccee4f4937ad6f7f16249fd748_2_1"/>
</fieldset>
<div class="indicator-container">
<span class="status unanswered" id="status_564da5ccee4f4937ad6f7f16249fd748_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 2" role="group"><div class="choicegroup capa_inputtype" id="inputtype_564da5ccee4f4937ad6f7f16249fd748_3_1">
<fieldset aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_3_1">
<legend id="564da5ccee4f4937ad6f7f16249fd748_3_1-legend" class="response-fieldset-legend field-group-hd"><img src="/assets/courseware/v1/a6081c7b1197c20b6c2d8f2485abc076/asset-v1:MITx+HST.953x+3T2020+type@asset+block/image4.png" alt="A ROC curve"/></legend>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_3_1" id="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_0" class="field-input input-radio" value="choice_0"/><label id="564da5ccee4f4937ad6f7f16249fd748_3_1-choice_0-label" for="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_0" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_3_1"> A model with low predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_3_1" id="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_1" class="field-input input-radio" value="choice_1"/><label id="564da5ccee4f4937ad6f7f16249fd748_3_1-choice_1-label" for="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_1" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_3_1"> A model with high predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_3_1" id="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_2" class="field-input input-radio" value="choice_2"/><label id="564da5ccee4f4937ad6f7f16249fd748_3_1-choice_2-label" for="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_2" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_3_1"> A model which outputs 1 or 0 at random
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_3_1" id="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_3" class="field-input input-radio" value="choice_3"/><label id="564da5ccee4f4937ad6f7f16249fd748_3_1-choice_3-label" for="input_564da5ccee4f4937ad6f7f16249fd748_3_1_choice_3" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_3_1"> A model which predicts the data perfectly
</label>
</div>
<span id="answer_564da5ccee4f4937ad6f7f16249fd748_3_1"/>
</fieldset>
<div class="indicator-container">
<span class="status unanswered" id="status_564da5ccee4f4937ad6f7f16249fd748_3_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 3" role="group"><div class="choicegroup capa_inputtype" id="inputtype_564da5ccee4f4937ad6f7f16249fd748_4_1">
<fieldset aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_4_1">
<legend id="564da5ccee4f4937ad6f7f16249fd748_4_1-legend" class="response-fieldset-legend field-group-hd"><img src="/assets/courseware/v1/f1c6e0aeed06e907e039d20ad8e3b147/asset-v1:MITx+HST.953x+3T2020+type@asset+block/image6.png" alt="A ROC curve"/></legend>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_4_1" id="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_0" class="field-input input-radio" value="choice_0"/><label id="564da5ccee4f4937ad6f7f16249fd748_4_1-choice_0-label" for="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_0" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_4_1"> A model with low predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_4_1" id="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_1" class="field-input input-radio" value="choice_1"/><label id="564da5ccee4f4937ad6f7f16249fd748_4_1-choice_1-label" for="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_1" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_4_1"> A model with high predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_4_1" id="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_2" class="field-input input-radio" value="choice_2"/><label id="564da5ccee4f4937ad6f7f16249fd748_4_1-choice_2-label" for="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_2" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_4_1"> A model which outputs 1 or 0 at random
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_4_1" id="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_3" class="field-input input-radio" value="choice_3"/><label id="564da5ccee4f4937ad6f7f16249fd748_4_1-choice_3-label" for="input_564da5ccee4f4937ad6f7f16249fd748_4_1_choice_3" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_4_1"> A model which predicts the data perfectly
</label>
</div>
<span id="answer_564da5ccee4f4937ad6f7f16249fd748_4_1"/>
</fieldset>
<div class="indicator-container">
<span class="status unanswered" id="status_564da5ccee4f4937ad6f7f16249fd748_4_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 4" role="group"><div class="choicegroup capa_inputtype" id="inputtype_564da5ccee4f4937ad6f7f16249fd748_5_1">
<fieldset aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_5_1">
<legend id="564da5ccee4f4937ad6f7f16249fd748_5_1-legend" class="response-fieldset-legend field-group-hd"><img src="/assets/courseware/v1/912cea3c36967b8e5fb197e2ba978a2c/asset-v1:MITx+HST.953x+3T2020+type@asset+block/image3.png" alt="A ROC curve"/></legend>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_5_1" id="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_0" class="field-input input-radio" value="choice_0"/><label id="564da5ccee4f4937ad6f7f16249fd748_5_1-choice_0-label" for="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_0" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_5_1"> A model with low predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_5_1" id="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_1" class="field-input input-radio" value="choice_1"/><label id="564da5ccee4f4937ad6f7f16249fd748_5_1-choice_1-label" for="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_1" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_5_1"> A model with high predictive power
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_5_1" id="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_2" class="field-input input-radio" value="choice_2"/><label id="564da5ccee4f4937ad6f7f16249fd748_5_1-choice_2-label" for="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_2" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_5_1"> A model which outputs 1 or 0 at random
</label>
</div>
<div class="field">
<input type="radio" name="input_564da5ccee4f4937ad6f7f16249fd748_5_1" id="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_3" class="field-input input-radio" value="choice_3"/><label id="564da5ccee4f4937ad6f7f16249fd748_5_1-choice_3-label" for="input_564da5ccee4f4937ad6f7f16249fd748_5_1_choice_3" class="response-label field-label label-inline" aria-describedby="status_564da5ccee4f4937ad6f7f16249fd748_5_1"> A model which predicts the data perfectly
</label>
</div>
<span id="answer_564da5ccee4f4937ad6f7f16249fd748_5_1"/>
</fieldset>
<div class="indicator-container">
<span class="status unanswered" id="status_564da5ccee4f4937ad6f7f16249fd748_5_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Matching ROC Curves" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_564da5ccee4f4937ad6f7f16249fd748" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_564da5ccee4f4937ad6f7f16249fd748">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="564da5ccee4f4937ad6f7f16249fd748-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="564da5ccee4f4937ad6f7f16249fd748-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="564da5ccee4f4937ad6f7f16249fd748-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="False">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@1bd42a6ace4e496aa7293c53aeced58c" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">Validation</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@51ffb40b2e754be6b150ccba8ae96290">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@51ffb40b2e754be6b150ccba8ae96290" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Validating your model</h3>
<p>As discussed in Data Analysis, validation is used to ensure that the model will perform similarly under different conditions - such as similar data from a new source. Several methods can be used to validate model performance:</p>
<ul>
<li>Only training on part of the dataset, keeping some back to validate data</li>
<li>K-fold cross-validation, explained below</li>
<li>External validation - verification on a data set from another source</li>
</ul>
<p>Of these methods, external validation is always the most rigorous and stringent way to check the validity of your model, because it addresses the possibility that there is something unrepresentative about your data source. However, it can sometimes be difficult to find another dataset which is similar enough for verification. Providing external validation is one reason why wide availability of multiple datasets is important.</p>
<p></p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@8038babfb1bc414b858c501b629ce11f">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@8038babfb1bc414b858c501b629ce11f" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>K-fold Cross-validation</h3>
<p>K-fold cross-validation is used to evaluate model accuracy in a similar way to a normal train/test split. Instead of using a single validation set, k-fold cross validation creates and tests on k different validation sets, using the procedure below:</p>
<ol>
<li>Shuffle the data</li>
<li>Split the data into k-different groups</li>
<li>For each group, use the other (k-1) groups as the training set and the kth group as the test set</li>
<li>Average evaluation scores across each fold</li>
</ol>
<p>K-fold cross validation is useful because it retains all of the data for model training (as opposed to setting some aside for validation) and it is less prone to bias than a single train/validation split.</p>
<p></p>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@abb49203635046ba9126cef8a3d86255">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@abb49203635046ba9126cef8a3d86255" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Exercise: Run K-fold Cross-Validation</h3>
<p><span style="color: #313131; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;">Please download the sensitivity analysis and model evaluation workshop R markdown file and the associated data set from </span><a href="https://github.com/criticaldata/hst953-edx/tree/master/2.10%20Sensitivity%20Analysis%20and%20Model%20Evaluation" target="[object Object]" style="color: #0075b4; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; text-decoration-line: none; transition: all 0.1s linear 0s;">GitHub</a><span style="color: #313131; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;"> and run the file, which runs k-fold cross validation on an SVM model for predicting heart disease. Once you have run the code, answer the questions below.</span></p>
<p></p>
</div>
</div>
<div class="vert vert-3" data-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@6b34ce0e9f634984ac7aba8a1adf9b52">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@6b34ce0e9f634984ac7aba8a1adf9b52" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="problem" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="True" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_6b34ce0e9f634984ac7aba8a1adf9b52" class="problems-wrapper" role="group"
aria-labelledby="6b34ce0e9f634984ac7aba8a1adf9b52-problem-title"
data-problem-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@6b34ce0e9f634984ac7aba8a1adf9b52" data-url="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@problem+block@6b34ce0e9f634984ac7aba8a1adf9b52/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="1"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="6b34ce0e9f634984ac7aba8a1adf9b52-problem-title" aria-describedby="block-v1:MITx+HST.953x+3T2020+type@problem+block@6b34ce0e9f634984ac7aba8a1adf9b52-problem-progress" tabindex="-1">
Question 2a: How many folds?
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.953x+3T2020+type@problem+block@6b34ce0e9f634984ac7aba8a1adf9b52-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><div id="formulaequationinput_6b34ce0e9f634984ac7aba8a1adf9b52_2_1" class="inputtype formulaequationinput">
<div class="unanswered">
<label class="problem-group-label" for="input_6b34ce0e9f634984ac7aba8a1adf9b52_2_1" id="label_6b34ce0e9f634984ac7aba8a1adf9b52_2_1">How many folds were used in the k-fold cross validation?</label>
<input type="text" name="input_6b34ce0e9f634984ac7aba8a1adf9b52_2_1" id="input_6b34ce0e9f634984ac7aba8a1adf9b52_2_1" data-input-id="6b34ce0e9f634984ac7aba8a1adf9b52_2_1" value="" aria-describedby="status_6b34ce0e9f634984ac7aba8a1adf9b52_2_1" size="20"/>
<span class="trailing_text" id="trailing_text_6b34ce0e9f634984ac7aba8a1adf9b52_2_1"/>
<span class="status unanswered" id="status_6b34ce0e9f634984ac7aba8a1adf9b52_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
<p id="answer_6b34ce0e9f634984ac7aba8a1adf9b52_2_1" class="answer"/>
<div id="input_6b34ce0e9f634984ac7aba8a1adf9b52_2_1_preview" class="equation">
\(\)
<img src="/static/images/spinner.bc34f953403f.gif" class="loading" alt="Loading"/>
</div>
</div>
<div class="script_placeholder" data-src="/static/js/capa/src/formula_equation_preview.b1967ab28c31.js"/>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Question 2a: How many folds?" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_6b34ce0e9f634984ac7aba8a1adf9b52" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_6b34ce0e9f634984ac7aba8a1adf9b52">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="6b34ce0e9f634984ac7aba8a1adf9b52-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="6b34ce0e9f634984ac7aba8a1adf9b52-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="6b34ce0e9f634984ac7aba8a1adf9b52-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="False">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
<div class="vert vert-4" data-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@8a97a5a5742d4c48acc344e17f566ecd">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@8a97a5a5742d4c48acc344e17f566ecd" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="problem" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="True" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_8a97a5a5742d4c48acc344e17f566ecd" class="problems-wrapper" role="group"
aria-labelledby="8a97a5a5742d4c48acc344e17f566ecd-problem-title"
data-problem-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@8a97a5a5742d4c48acc344e17f566ecd" data-url="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@problem+block@8a97a5a5742d4c48acc344e17f566ecd/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="1"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="8a97a5a5742d4c48acc344e17f566ecd-problem-title" aria-describedby="block-v1:MITx+HST.953x+3T2020+type@problem+block@8a97a5a5742d4c48acc344e17f566ecd-problem-progress" tabindex="-1">
Question 2b: How much data in each validation set?
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.953x+3T2020+type@problem+block@8a97a5a5742d4c48acc344e17f566ecd-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><div id="formulaequationinput_8a97a5a5742d4c48acc344e17f566ecd_2_1" class="inputtype formulaequationinput">
<div class="unanswered">
<label class="problem-group-label" for="input_8a97a5a5742d4c48acc344e17f566ecd_2_1" id="label_8a97a5a5742d4c48acc344e17f566ecd_2_1">What proportion of the dataset was used as a validation set after each fold?
Type your answer as a decimal, so if you think half the dataset was used you would answer '0.5'</label>
<input type="text" name="input_8a97a5a5742d4c48acc344e17f566ecd_2_1" id="input_8a97a5a5742d4c48acc344e17f566ecd_2_1" data-input-id="8a97a5a5742d4c48acc344e17f566ecd_2_1" value="" aria-describedby="status_8a97a5a5742d4c48acc344e17f566ecd_2_1" size="20"/>
<span class="trailing_text" id="trailing_text_8a97a5a5742d4c48acc344e17f566ecd_2_1"/>
<span class="status unanswered" id="status_8a97a5a5742d4c48acc344e17f566ecd_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
<p id="answer_8a97a5a5742d4c48acc344e17f566ecd_2_1" class="answer"/>
<div id="input_8a97a5a5742d4c48acc344e17f566ecd_2_1_preview" class="equation">
\(\)
<img src="/static/images/spinner.bc34f953403f.gif" class="loading" alt="Loading"/>
</div>
</div>
<div class="script_placeholder" data-src="/static/js/capa/src/formula_equation_preview.b1967ab28c31.js"/>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Question 2b: How much data in each validation set?" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_8a97a5a5742d4c48acc344e17f566ecd" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_8a97a5a5742d4c48acc344e17f566ecd">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="8a97a5a5742d4c48acc344e17f566ecd-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="8a97a5a5742d4c48acc344e17f566ecd-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="8a97a5a5742d4c48acc344e17f566ecd-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="False">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
<div class="vert vert-5" data-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@f6c8401362f3489a86a35af716b4dc45">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@f6c8401362f3489a86a35af716b4dc45" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="problem" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="True" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_f6c8401362f3489a86a35af716b4dc45" class="problems-wrapper" role="group"
aria-labelledby="f6c8401362f3489a86a35af716b4dc45-problem-title"
data-problem-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@f6c8401362f3489a86a35af716b4dc45" data-url="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@problem+block@f6c8401362f3489a86a35af716b4dc45/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="1"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="f6c8401362f3489a86a35af716b4dc45-problem-title" aria-describedby="block-v1:MITx+HST.953x+3T2020+type@problem+block@f6c8401362f3489a86a35af716b4dc45-problem-progress" tabindex="-1">
Question 3: Hyperparameter tuning
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.953x+3T2020+type@problem+block@f6c8401362f3489a86a35af716b4dc45-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><p>You can use this template as a guide to the simple editor markdown and OLX markup to use for multiple choice problems. Edit this component to replace this template with your own assessment.</p>
<div class="choicegroup capa_inputtype" id="inputtype_f6c8401362f3489a86a35af716b4dc45_2_1">
<fieldset aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1">
<legend id="f6c8401362f3489a86a35af716b4dc45_2_1-legend" class="response-fieldset-legend field-group-hd">Which value of C had the highest accuracy in the validation set?</legend>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_0" class="field-input input-radio" value="choice_0"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_0-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_0" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 0.001
</label>
</div>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_1" class="field-input input-radio" value="choice_1"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_1-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_1" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 0.01
</label>
</div>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_2" class="field-input input-radio" value="choice_2"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_2-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_2" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 0.05
</label>
</div>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_3" class="field-input input-radio" value="choice_3"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_3-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_3" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 0.1
</label>
</div>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_4" class="field-input input-radio" value="choice_4"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_4-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_4" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 0.5
</label>
</div>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_5" class="field-input input-radio" value="choice_5"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_5-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_5" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 1
</label>
</div>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_6" class="field-input input-radio" value="choice_6"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_6-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_6" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 5
</label>
</div>
<div class="field">
<input type="radio" name="input_f6c8401362f3489a86a35af716b4dc45_2_1" id="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_7" class="field-input input-radio" value="choice_7"/><label id="f6c8401362f3489a86a35af716b4dc45_2_1-choice_7-label" for="input_f6c8401362f3489a86a35af716b4dc45_2_1_choice_7" class="response-label field-label label-inline" aria-describedby="status_f6c8401362f3489a86a35af716b4dc45_2_1"> 20
</label>
</div>
<span id="answer_f6c8401362f3489a86a35af716b4dc45_2_1"/>
</fieldset>
<div class="indicator-container">
<span class="status unanswered" id="status_f6c8401362f3489a86a35af716b4dc45_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Question 3: Hyperparameter tuning" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_f6c8401362f3489a86a35af716b4dc45" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_f6c8401362f3489a86a35af716b4dc45">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="f6c8401362f3489a86a35af716b4dc45-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="f6c8401362f3489a86a35af716b4dc45-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="f6c8401362f3489a86a35af716b4dc45-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="False">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
<div class="vert vert-6" data-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@1bb443b79a3e45328323c31009d5f638">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@1bb443b79a3e45328323c31009d5f638" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="problem" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="True" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_1bb443b79a3e45328323c31009d5f638" class="problems-wrapper" role="group"
aria-labelledby="1bb443b79a3e45328323c31009d5f638-problem-title"
data-problem-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@1bb443b79a3e45328323c31009d5f638" data-url="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@problem+block@1bb443b79a3e45328323c31009d5f638/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="1"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="1bb443b79a3e45328323c31009d5f638-problem-title" aria-describedby="block-v1:MITx+HST.953x+3T2020+type@problem+block@1bb443b79a3e45328323c31009d5f638-problem-progress" tabindex="-1">
Question 4: Specificity
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.953x+3T2020+type@problem+block@1bb443b79a3e45328323c31009d5f638-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><div id="formulaequationinput_1bb443b79a3e45328323c31009d5f638_2_1" class="inputtype formulaequationinput">
<div class="unanswered">
<label class="problem-group-label" for="input_1bb443b79a3e45328323c31009d5f638_2_1" id="label_1bb443b79a3e45328323c31009d5f638_2_1">What was the specificity of the model in the test set?</label>
<input type="text" name="input_1bb443b79a3e45328323c31009d5f638_2_1" id="input_1bb443b79a3e45328323c31009d5f638_2_1" data-input-id="1bb443b79a3e45328323c31009d5f638_2_1" value="" aria-describedby="status_1bb443b79a3e45328323c31009d5f638_2_1" size="20"/>
<span class="trailing_text" id="trailing_text_1bb443b79a3e45328323c31009d5f638_2_1"/>
<span class="status unanswered" id="status_1bb443b79a3e45328323c31009d5f638_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
<p id="answer_1bb443b79a3e45328323c31009d5f638_2_1" class="answer"/>
<div id="input_1bb443b79a3e45328323c31009d5f638_2_1_preview" class="equation">
\(\)
<img src="/static/images/spinner.bc34f953403f.gif" class="loading" alt="Loading"/>
</div>
</div>
<div class="script_placeholder" data-src="/static/js/capa/src/formula_equation_preview.b1967ab28c31.js"/>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Question 4: Specificity" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_1bb443b79a3e45328323c31009d5f638" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_1bb443b79a3e45328323c31009d5f638">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="1bb443b79a3e45328323c31009d5f638-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="1bb443b79a3e45328323c31009d5f638-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="1bb443b79a3e45328323c31009d5f638-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="False">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
<div class="vert vert-7" data-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@2704b0e7c77a4b0a89159352d15bd61d">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@2704b0e7c77a4b0a89159352d15bd61d" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="problem" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="True" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_2704b0e7c77a4b0a89159352d15bd61d" class="problems-wrapper" role="group"
aria-labelledby="2704b0e7c77a4b0a89159352d15bd61d-problem-title"
data-problem-id="block-v1:MITx+HST.953x+3T2020+type@problem+block@2704b0e7c77a4b0a89159352d15bd61d" data-url="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@problem+block@2704b0e7c77a4b0a89159352d15bd61d/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="5"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="2704b0e7c77a4b0a89159352d15bd61d-problem-title" aria-describedby="block-v1:MITx+HST.953x+3T2020+type@problem+block@2704b0e7c77a4b0a89159352d15bd61d-problem-progress" tabindex="-1">
Question 5: Rank the validation methods
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.953x+3T2020+type@problem+block@2704b0e7c77a4b0a89159352d15bd61d-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><div class="inputtype option-input ">
<label class="problem-group-label" for="input_2704b0e7c77a4b0a89159352d15bd61d_2_1" id="label_2704b0e7c77a4b0a89159352d15bd61d_2_1">Rank these different validation methods in order of least to most robust validation:</label>
<select name="input_2704b0e7c77a4b0a89159352d15bd61d_2_1" id="input_2704b0e7c77a4b0a89159352d15bd61d_2_1" aria-describedby="status_2704b0e7c77a4b0a89159352d15bd61d_2_1">
<option value="option_2704b0e7c77a4b0a89159352d15bd61d_2_1_dummy_default">Select an option</option>
<option value="K-fold cross validation"> K-fold cross validation</option>
<option value="Validating on training set"> Validating on training set</option>
<option value="Unseen data from a different source"> Unseen data from a different source</option>
<option value="Validating on validation set (the set used to choose model parameters)"> Validating on validation set (the set used to choose model parameters)</option>
<option value="Unseen data from the same source - also known as a test set"> Unseen data from the same source - also known as a test set</option>
</select>
<div class="indicator-container">
<span class="status unanswered" id="status_2704b0e7c77a4b0a89159352d15bd61d_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
<p class="answer" id="answer_2704b0e7c77a4b0a89159352d15bd61d_2_1"/>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 2" role="group"><div class="inputtype option-input ">
<select name="input_2704b0e7c77a4b0a89159352d15bd61d_3_1" id="input_2704b0e7c77a4b0a89159352d15bd61d_3_1" aria-describedby="status_2704b0e7c77a4b0a89159352d15bd61d_3_1">
<option value="option_2704b0e7c77a4b0a89159352d15bd61d_3_1_dummy_default">Select an option</option>
<option value="K-fold cross validation"> K-fold cross validation</option>
<option value="Validating on training set"> Validating on training set</option>
<option value="Unseen data from a different source"> Unseen data from a different source</option>
<option value="Validating on validation set (the set used to choose model parameters)"> Validating on validation set (the set used to choose model parameters)</option>
<option value="Unseen data from the same source - also known as a test set"> Unseen data from the same source - also known as a test set</option>
</select>
<div class="indicator-container">
<span class="status unanswered" id="status_2704b0e7c77a4b0a89159352d15bd61d_3_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
<p class="answer" id="answer_2704b0e7c77a4b0a89159352d15bd61d_3_1"/>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 3" role="group"><div class="inputtype option-input ">
<select name="input_2704b0e7c77a4b0a89159352d15bd61d_4_1" id="input_2704b0e7c77a4b0a89159352d15bd61d_4_1" aria-describedby="status_2704b0e7c77a4b0a89159352d15bd61d_4_1">
<option value="option_2704b0e7c77a4b0a89159352d15bd61d_4_1_dummy_default">Select an option</option>
<option value="K-fold cross validation"> K-fold cross validation</option>
<option value="Validating on training set"> Validating on training set</option>
<option value="Unseen data from a different source"> Unseen data from a different source</option>
<option value="Validating on validation set (the set used to choose model parameters)"> Validating on validation set (the set used to choose model parameters)</option>
<option value="Unseen data from the same source - also known as a test set"> Unseen data from the same source - also known as a test set</option>
</select>
<div class="indicator-container">
<span class="status unanswered" id="status_2704b0e7c77a4b0a89159352d15bd61d_4_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
<p class="answer" id="answer_2704b0e7c77a4b0a89159352d15bd61d_4_1"/>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 4" role="group"><div class="inputtype option-input ">
<select name="input_2704b0e7c77a4b0a89159352d15bd61d_5_1" id="input_2704b0e7c77a4b0a89159352d15bd61d_5_1" aria-describedby="status_2704b0e7c77a4b0a89159352d15bd61d_5_1">
<option value="option_2704b0e7c77a4b0a89159352d15bd61d_5_1_dummy_default">Select an option</option>
<option value="K-fold cross validation"> K-fold cross validation</option>
<option value="Validating on training set"> Validating on training set</option>
<option value="Unseen data from a different source"> Unseen data from a different source</option>
<option value="Validating on validation set (the set used to choose model parameters)"> Validating on validation set (the set used to choose model parameters)</option>
<option value="Unseen data from the same source - also known as a test set"> Unseen data from the same source - also known as a test set</option>
</select>
<div class="indicator-container">
<span class="status unanswered" id="status_2704b0e7c77a4b0a89159352d15bd61d_5_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
<p class="answer" id="answer_2704b0e7c77a4b0a89159352d15bd61d_5_1"/>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 5" role="group"><div class="inputtype option-input ">
<select name="input_2704b0e7c77a4b0a89159352d15bd61d_6_1" id="input_2704b0e7c77a4b0a89159352d15bd61d_6_1" aria-describedby="status_2704b0e7c77a4b0a89159352d15bd61d_6_1">
<option value="option_2704b0e7c77a4b0a89159352d15bd61d_6_1_dummy_default">Select an option</option>
<option value="K-fold cross validation"> K-fold cross validation</option>
<option value="Validating on training set"> Validating on training set</option>
<option value="Unseen data from a different source"> Unseen data from a different source</option>
<option value="Validating on validation set (the set used to choose model parameters)"> Validating on validation set (the set used to choose model parameters)</option>
<option value="Unseen data from the same source - also known as a test set"> Unseen data from the same source - also known as a test set</option>
</select>
<div class="indicator-container">
<span class="status unanswered" id="status_2704b0e7c77a4b0a89159352d15bd61d_6_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
<p class="answer" id="answer_2704b0e7c77a4b0a89159352d15bd61d_6_1"/>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Question 5: Rank the validation methods" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_2704b0e7c77a4b0a89159352d15bd61d" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_2704b0e7c77a4b0a89159352d15bd61d">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="2704b0e7c77a4b0a89159352d15bd61d-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="2704b0e7c77a4b0a89159352d15bd61d-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="2704b0e7c77a4b0a89159352d15bd61d-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="False">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@590b29119a6440d98fc9240e46f2e468" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">Sensitivity Analysis</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@0c2572e70ea840eeabae66657f111a9e">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@0c2572e70ea840eeabae66657f111a9e" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Sensitivity analysis</h3>
<p>The creation of any model requires assumptions and, sometimes arbitrary, decisions on parameter values. The purpose of sensitivity analysis is to assess if the model results are robust to these decisions, or small changes lead to large fluctuations in results. Results that remain consistent throughout sensitivity analysis are more likely to generalize to different datasets.</p>
<p>What sensitivity analysis you perform depends on the assumptions used to construct the model and the structure of your data. Common areas to analyse include:</p>
<ul>
<li>Which features are important to collect? If there are unimportant features could these be removed from the model to reduce computation time and the burden of collection?</li>
<li>How robust is the model to noise in the data?</li>
<li>How are the results affected when you change parameters in your model, such as the stringency of propensity score matching?</li>
<li>How does the cohort affect the results? Are the results replicable with different inclusion criteria? Do your results hold in different subgroups?</li>
</ul>
<p>The best test of your results is to replicate them with many models and using different data sources.</p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1">
<div class="xblock xblock-public_view xblock-public_view-video xmodule_display xmodule_VideoBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="video" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Video"}
</script>
<h3 class="hd hd-2">Sensitivity Analysis Example</h3>
<div
id="video_08d7de4a3b36405e9eddb9e5852536b1"
class="video closed"
data-metadata='{"saveStateUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1/handler/xmodule_handler/save_user_state", "lmsRootURL": "https://openlearninglibrary.mit.edu", "publishCompletionUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1/handler/publish_completion", "streams": "1.00:NFExz078dMo", "duration": 0.0, "recordedYoutubeIsAvailable": true, "transcriptAvailableTranslationsUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1/handler/transcript/available_translations", "captionDataDir": null, "ytApiUrl": "https://www.youtube.com/iframe_api", "speed": null, "end": 0.0, "completionPercentage": 0.95, "autoAdvance": false, "transcriptLanguage": "en", "prioritizeHls": false, "autohideHtml5": false, "ytTestTimeout": 1500, "transcriptLanguages": {"en": "English"}, "savedVideoPosition": 0.0, "sources": [], "completionEnabled": false, "saveStateEnabled": false, "generalSpeed": 1.0, "autoplay": false, "poster": null, "showCaptions": "true", "transcriptTranslationUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1/handler/transcript/translation/__lang__", "ytMetadataEndpoint": "", "start": 0.0}'
data-bumper-metadata='null'
data-autoadvance-enabled="False"
data-poster='null'
tabindex="-1"
>
<div class="focus_grabber first"></div>
<div class="tc-wrapper">
<div class="video-wrapper">
<span tabindex="0" class="spinner" aria-hidden="false" aria-label="Loading video player"></span>
<span tabindex="-1" class="btn-play fa fa-youtube-play fa-2x is-hidden" aria-hidden="true" aria-label="Play video"></span>
<div class="video-player-pre"></div>
<div class="video-player">
<div id="08d7de4a3b36405e9eddb9e5852536b1"></div>
<h4 class="hd hd-4 video-error is-hidden">No playable video sources found.</h4>
<h4 class="hd hd-4 video-hls-error is-hidden">
Your browser does not support this video format. Try using a different browser.
</h4>
</div>
<div class="video-player-post"></div>
<div class="closed-captions"></div>
<div class="video-controls is-hidden">
<div>
<div class="vcr"><div class="vidtime">0:00 / 0:00</div></div>
<div class="secondary-controls"></div>
</div>
</div>
</div>
</div>
<div class="focus_grabber last"></div>
<h3 class="hd hd-4 downloads-heading sr" id="video-download-transcripts_08d7de4a3b36405e9eddb9e5852536b1">Downloads and transcripts</h3>
<div class="wrapper-downloads" role="region" aria-labelledby="video-download-transcripts_08d7de4a3b36405e9eddb9e5852536b1">
<div class="wrapper-download-transcripts">
<h4 class="hd hd-5">Transcripts</h4>
<ul class="list-download-transcripts">
<li class="transcript-option">
<a class="btn btn-link" href="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1/handler/transcript/download" data-value="srt">Download SubRip (.srt) file</a>
</li>
<li class="transcript-option">
<a class="btn btn-link" href="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@08d7de4a3b36405e9eddb9e5852536b1/handler/transcript/download" data-value="txt">Download Text (.txt) file</a>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@vertical+block@f3c2f155ab3d4f78b764f24fd189ed6b" data-init="VerticalStudentView" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="vertical" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<h2 class="hd hd-2 unit-title">Conclusion</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064">
<div class="xblock xblock-public_view xblock-public_view-video xmodule_display xmodule_VideoBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="video" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Video"}
</script>
<h3 class="hd hd-2">Final Thoughts</h3>
<div
id="video_d641a36821114070bf2ab15e5ab01064"
class="video closed"
data-metadata='{"saveStateUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064/handler/xmodule_handler/save_user_state", "lmsRootURL": "https://openlearninglibrary.mit.edu", "publishCompletionUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064/handler/publish_completion", "streams": "1.00:WMD-fDSTKIg", "duration": 0.0, "recordedYoutubeIsAvailable": true, "transcriptAvailableTranslationsUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064/handler/transcript/available_translations", "captionDataDir": null, "ytApiUrl": "https://www.youtube.com/iframe_api", "speed": null, "end": 0.0, "completionPercentage": 0.95, "autoAdvance": false, "transcriptLanguage": "en", "prioritizeHls": false, "autohideHtml5": false, "ytTestTimeout": 1500, "transcriptLanguages": {"en": "English"}, "savedVideoPosition": 0.0, "sources": [], "completionEnabled": false, "saveStateEnabled": false, "generalSpeed": 1.0, "autoplay": false, "poster": null, "showCaptions": "true", "transcriptTranslationUrl": "/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064/handler/transcript/translation/__lang__", "ytMetadataEndpoint": "", "start": 0.0}'
data-bumper-metadata='null'
data-autoadvance-enabled="False"
data-poster='null'
tabindex="-1"
>
<div class="focus_grabber first"></div>
<div class="tc-wrapper">
<div class="video-wrapper">
<span tabindex="0" class="spinner" aria-hidden="false" aria-label="Loading video player"></span>
<span tabindex="-1" class="btn-play fa fa-youtube-play fa-2x is-hidden" aria-hidden="true" aria-label="Play video"></span>
<div class="video-player-pre"></div>
<div class="video-player">
<div id="d641a36821114070bf2ab15e5ab01064"></div>
<h4 class="hd hd-4 video-error is-hidden">No playable video sources found.</h4>
<h4 class="hd hd-4 video-hls-error is-hidden">
Your browser does not support this video format. Try using a different browser.
</h4>
</div>
<div class="video-player-post"></div>
<div class="closed-captions"></div>
<div class="video-controls is-hidden">
<div>
<div class="vcr"><div class="vidtime">0:00 / 0:00</div></div>
<div class="secondary-controls"></div>
</div>
</div>
</div>
</div>
<div class="focus_grabber last"></div>
<h3 class="hd hd-4 downloads-heading sr" id="video-download-transcripts_d641a36821114070bf2ab15e5ab01064">Downloads and transcripts</h3>
<div class="wrapper-downloads" role="region" aria-labelledby="video-download-transcripts_d641a36821114070bf2ab15e5ab01064">
<div class="wrapper-download-transcripts">
<h4 class="hd hd-5">Transcripts</h4>
<ul class="list-download-transcripts">
<li class="transcript-option">
<a class="btn btn-link" href="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064/handler/transcript/download" data-value="srt">Download SubRip (.srt) file</a>
</li>
<li class="transcript-option">
<a class="btn btn-link" href="/courses/course-v1:MITx+HST.953x+3T2020/xblock/block-v1:MITx+HST.953x+3T2020+type@video+block@d641a36821114070bf2ab15e5ab01064/handler/transcript/download" data-value="txt">Download Text (.txt) file</a>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@3c6aca520c294960865b623c718adb84">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@3c6aca520c294960865b623c718adb84" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Take home messages</h3>
<ul>
<li>alidation and sensitivity analyses test the robustness of the model assumptions and are a key step in the modeling process.</li>
<li>The key principle of both these analyses is to vary the model assumptions and observe how the model responds</li>
<li>Failing the validation and sensitivity analyses might require the researcher to start with a new model</li>
</ul>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.953x+3T2020+type@html+block@b76fc0a56ce6443ab1c06f295a7f18d5">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-usage-id="block-v1:MITx+HST.953x+3T2020+type@html+block@b76fc0a56ce6443ab1c06f295a7f18d5" data-init="XBlockToXModuleShim" data-graded="False" data-request-token="7f4b9b5843c111ef8f120e08775edbcd" data-block-type="html" data-runtime-version="1" data-course-id="course-v1:MITx+HST.953x+3T2020" data-has-score="False" data-runtime-class="LmsRuntime">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h3>Conclusion</h3>
<p>The analysis of observational health data carries the inherent limitation of unmeasured confounding. After model development and primary analysis, an important step is to confirm a model’s performance. Validation can be used to check that the model is an appropriate fit for the data and is likely to perform similarly in other cohorts. Sensitivity analysis can be used to interrogate inherent assumptions of the primary analysis. If a model fails these tests, this could indicate that a new approach is required. </p>
<p>When performed adequately these additional steps help improve the robustness of the overall analysis and aid the investigator in making meaningful inferences from observational health data. </p>
<p></p>
</div>
</div>
</div>
</div>
© All Rights Reserved