<div class="xblock xblock-public_view xblock-public_view-vertical" data-init="VerticalStudentView" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@vertical+block@dc930fdf91b649a49f7cae14d7e83bf3" data-runtime-class="LmsRuntime" data-block-type="vertical" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<h2 class="hd hd-2 unit-title">The Rationale</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@26a20a49ef6d4fe58fbbeaf9136b0c80">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@26a20a49ef6d4fe58fbbeaf9136b0c80" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Why evaluate health information systems?</h4>
<p>Health information systems (HIS) are becoming increasingly widely used worldwide including in some of the lowest income countries such as Haiti, Rwanda and Cambodia.As discussed in earlier chapters there are many good reasons why these systems are growing in use, both in terms of perceived need, improvements in software and in the availability of smaller, cheaper and more capable hardware that uses a lot less power. However the deployment and use of HIS should be based on real information needs and on solid evidence that their use improves aspects of healthcare, and patient health more generally. This requires rigorous evaluation studies and publication of evidence to allow development of robust guidance for decision makers. Evaluation studies need to assess a range of parameters including system performance, usability and stability and crucially data quality, as well as the impact of the systems on patient care processes and ideally clinical outcomes, and also examine the costs and cost effectiveness of systems. In this section we will discuss the types of evaluations that can performed, the different evaluation methods with their strengths and weaknesses, and give examples of some key evaluation results.</p>
<p>An important point to note is that most HIS do not undergo evaluation at all stages of development and frequently are not evaluated at all. If the technology used is mature then there may not be a requirement to carry out the evaluation of the earlier stages of software development, but if there are significant changes it is important to ensure they have been implemented correctly and assess their impact. Evaluations do not need to be large or complex to yield valuable insights and data that can help the development and implementation process and inform decision-makers. A goal of this section is to illustrate that evaluation can be carried out by people with a wide range of backgrounds and in a range of environments without necessarily requiring large resources.</p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@de68aae106524c6f9cde775f85883009">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@de68aae106524c6f9cde775f85883009" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<p><span style="font-family: 'Open Sans', Verdana, Arial, Helvetica, sans-serif;">The videos in this unit are presented by:</span></p>
<p><span style="font-family: 'Open Sans', Verdana, Arial, Helvetica, sans-serif;">Hamish Fraser MBChB</span><br style="text-rendering: optimizeLegibility; line-height: 1.4em; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;" /><span style="font-family: 'Open Sans', Verdana, Arial, Helvetica, sans-serif;">Professor</span><br style="text-rendering: optimizeLegibility; line-height: 1.4em; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;" /><span style="font-family: 'Open Sans', Verdana, Arial, Helvetica, sans-serif;">Leeds Institute of Health Sciences</span><br style="text-rendering: optimizeLegibility; line-height: 1.4em; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;" /><span style="font-family: 'Open Sans', Verdana, Arial, Helvetica, sans-serif;">University of Leeds</span></p>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.936x+1T2019+type@video+block@9638cef52f504b84a5503f262a1f457c">
<div class="xblock xblock-public_view xblock-public_view-video xmodule_display xmodule_VideoBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@video+block@9638cef52f504b84a5503f262a1f457c" data-runtime-class="LmsRuntime" data-block-type="video" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Video"}
</script>
<h3 class="hd hd-2">Why Evaluate Health Information Systems?</h3>
<div
id="video_9638cef52f504b84a5503f262a1f457c"
class="video closed"
data-metadata='{"speed": null, "saveStateEnabled": false, "streams": "1.00:yuGpkvWAsqo", "savedVideoPosition": 0.0, "ytTestTimeout": 1500, "generalSpeed": 1.0, "poster": "https://mitx-edx-video-meta-storage.s3.amazonaws.com/media/video-images/902203fa48c04e5fb7d4d562ac17b47b.jpeg?Expires=1714136655&AWSAccessKeyId=ASIA4QDFFSURSAMMO4WA&Signature=JR0FoEA5X5M93jYmmlRKzDYdCTw%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEHUaCXVzLWVhc3QtMSJHMEUCIDgpsiMUMvtU%2BUkvEvgdEEjHJ8vgnN4V4h3%2F9z4USHLPAiEA3vw2OU%2F9k2EBtFnNV7NnN4CDaBu9V7uh2pez5%2FPXgL0qwwUIvv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARADGgw4NTkyMDU2Mzc0MTEiDDFbNXwLSS8FzBirSSqXBVXRa5MzUWeMZQl%2B4KenRrtNMSSax%2FGns%2Bxx4JigP58Cx212DKqpvpXjBEOXYH1g187Bx5bii9w%2Bm5btq7rahKFjFWGZE3ZESWoBZHDareII25IjkDCPmGskiLFgeB7w1W1FnpqvLdZ2iguKnVUypb8THpgei990qCPBIXyCVjfJgxgEdcR%2B4AJhz3Ac%2BKKkXyhbB9Ob7xt%2B6lBtHRSPGZ%2B2iY1b37weOXi07Oy%2BwAJrLM%2FYP26BGswnMYG2V0ynlLkIVsS4b0Ig0ehvADts4L8yAvaTBsanYkAJyl8LIhdsLIyKm3WeqeHl2rXPii17Br8xVsC6lnnbnmfiQBzmwkxRKk06LSUSD8fIzTCzwcJaYSuvuPqcn%2Fq0LL49l6NyhoqPb3zVR0yE77RXChS0%2BFRN7s3Y8SzNU69LPsAzHlZZT97tiI%2BeQdP%2Fogd9voUoF5enSEeuhz808%2F0l79SytPW8vxlawF%2FxhLJPxkHzsEBNAbWE08Olw3Xqfk2QhYtc822S51Bzy45OwzxHTOvQ3GKZebI298zzHTUJaUu37f51zPu7yjgZ4Xl3xMTFkBQS%2FSKufmdWobRnDAhNN0YiVlF2vwlCcjSzOq3hIPaTeEIuf3dvhfk4Af%2F%2FjWJ5ZikVwnY1O%2FFcwTG9qvNVLhGqFblxM5PGr3cBhHm4NAAukqhYtSukoFyRXiyst7kvNGWw4s886avaJFw%2BGRsyvdSm1j2VvgiZR0R5RfL1Sjs68VbpE0Pogft8x%2FGzqApu8idKWnLRBPYdeH3K2bm0TDbXuOEkU1mdpCyFIBm7%2B2pNbx0HGQlTW3tAbiHWhwVdy0p%2FEt7tkDmIe66SpStA8uPBo36AD00nlEips7jxZEjnAYF4w38DVAeL5jCQvK6xBjqxAZ4Z1X3hUyfov8klf0dNokOaYTstnfamuDerU%2BSEBtS42PFpQSNpNipzl8%2F1oad2HbBezNm60UMih2qM1gYDCnN8KAUIyVuJQMOd2Ps3GW31adoWWu420CMrCLJP%2F3riLr1WhTEbVyq5EIghsVyToqVfYg9oTj1yIUarKF1RVlBZt%2FRI8Bq1tsmKOQuocfOy9bXgQbaPODrbZGsOEcQTn8FQkajsRJ1nG7OzwOCYUndiJw%3D%3D", "publishCompletionUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@9638cef52f504b84a5503f262a1f457c/handler/publish_completion", "completionEnabled": false, "autoplay": false, "completionPercentage": 0.95, "captionDataDir": null, "duration": 0.0, "saveStateUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@9638cef52f504b84a5503f262a1f457c/handler/xmodule_handler/save_user_state", "ytApiUrl": "https://www.youtube.com/iframe_api", "start": 0.0, "transcriptLanguage": "en", "lmsRootURL": "https://openlearninglibrary.mit.edu", "prioritizeHls": false, "autoAdvance": false, "transcriptLanguages": {"en": "English"}, "autohideHtml5": false, "transcriptTranslationUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@9638cef52f504b84a5503f262a1f457c/handler/transcript/translation/__lang__", "transcriptAvailableTranslationsUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@9638cef52f504b84a5503f262a1f457c/handler/transcript/available_translations", "ytMetadataEndpoint": "", "recordedYoutubeIsAvailable": true, "sources": ["https://d2f1egay8yehza.cloudfront.net/MITHST932017-V004400_DTH.mp4", "https://d2f1egay8yehza.cloudfront.net/MITHST932017-V004400/MITHST932017-V004400.m3u8"], "showCaptions": "true", "end": 0.0}'
data-bumper-metadata='null'
data-autoadvance-enabled="False"
data-poster='null'
tabindex="-1"
>
<div class="focus_grabber first"></div>
<div class="tc-wrapper">
<div class="video-wrapper">
<span tabindex="0" class="spinner" aria-hidden="false" aria-label="Loading video player"></span>
<span tabindex="-1" class="btn-play fa fa-youtube-play fa-2x is-hidden" aria-hidden="true" aria-label="Play video"></span>
<div class="video-player-pre"></div>
<div class="video-player">
<div id="9638cef52f504b84a5503f262a1f457c"></div>
<h4 class="hd hd-4 video-error is-hidden">No playable video sources found.</h4>
<h4 class="hd hd-4 video-hls-error is-hidden">
Your browser does not support this video format. Try using a different browser.
</h4>
</div>
<div class="video-player-post"></div>
<div class="closed-captions"></div>
<div class="video-controls is-hidden">
<div>
<div class="vcr"><div class="vidtime">0:00 / 0:00</div></div>
<div class="secondary-controls"></div>
</div>
</div>
</div>
</div>
<div class="focus_grabber last"></div>
</div>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-init="VerticalStudentView" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@vertical+block@4cd48651f7224d2099d4acb0de9c12fa" data-runtime-class="LmsRuntime" data-block-type="vertical" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<h2 class="hd hd-2 unit-title">Types of Evaluation</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.936x+1T2019+type@video+block@aeceba23dbb14725a3e63c3b79bded6f">
<div class="xblock xblock-public_view xblock-public_view-video xmodule_display xmodule_VideoBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@video+block@aeceba23dbb14725a3e63c3b79bded6f" data-runtime-class="LmsRuntime" data-block-type="video" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Video"}
</script>
<h3 class="hd hd-2">Types of Evaluation</h3>
<div
id="video_aeceba23dbb14725a3e63c3b79bded6f"
class="video closed"
data-metadata='{"speed": null, "saveStateEnabled": false, "streams": "1.00:mR08pJ4Z28s", "savedVideoPosition": 0.0, "ytTestTimeout": 1500, "generalSpeed": 1.0, "poster": "https://mitx-edx-video-meta-storage.s3.amazonaws.com/media/video-images/7ff598a06d2f45dcb46231e434c256d4.jpeg?Expires=1714136655&AWSAccessKeyId=ASIA4QDFFSURSAMMO4WA&Signature=bkfHJkHEbTWoIW6Ja9ckcqpt%2FJs%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEHUaCXVzLWVhc3QtMSJHMEUCIDgpsiMUMvtU%2BUkvEvgdEEjHJ8vgnN4V4h3%2F9z4USHLPAiEA3vw2OU%2F9k2EBtFnNV7NnN4CDaBu9V7uh2pez5%2FPXgL0qwwUIvv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARADGgw4NTkyMDU2Mzc0MTEiDDFbNXwLSS8FzBirSSqXBVXRa5MzUWeMZQl%2B4KenRrtNMSSax%2FGns%2Bxx4JigP58Cx212DKqpvpXjBEOXYH1g187Bx5bii9w%2Bm5btq7rahKFjFWGZE3ZESWoBZHDareII25IjkDCPmGskiLFgeB7w1W1FnpqvLdZ2iguKnVUypb8THpgei990qCPBIXyCVjfJgxgEdcR%2B4AJhz3Ac%2BKKkXyhbB9Ob7xt%2B6lBtHRSPGZ%2B2iY1b37weOXi07Oy%2BwAJrLM%2FYP26BGswnMYG2V0ynlLkIVsS4b0Ig0ehvADts4L8yAvaTBsanYkAJyl8LIhdsLIyKm3WeqeHl2rXPii17Br8xVsC6lnnbnmfiQBzmwkxRKk06LSUSD8fIzTCzwcJaYSuvuPqcn%2Fq0LL49l6NyhoqPb3zVR0yE77RXChS0%2BFRN7s3Y8SzNU69LPsAzHlZZT97tiI%2BeQdP%2Fogd9voUoF5enSEeuhz808%2F0l79SytPW8vxlawF%2FxhLJPxkHzsEBNAbWE08Olw3Xqfk2QhYtc822S51Bzy45OwzxHTOvQ3GKZebI298zzHTUJaUu37f51zPu7yjgZ4Xl3xMTFkBQS%2FSKufmdWobRnDAhNN0YiVlF2vwlCcjSzOq3hIPaTeEIuf3dvhfk4Af%2F%2FjWJ5ZikVwnY1O%2FFcwTG9qvNVLhGqFblxM5PGr3cBhHm4NAAukqhYtSukoFyRXiyst7kvNGWw4s886avaJFw%2BGRsyvdSm1j2VvgiZR0R5RfL1Sjs68VbpE0Pogft8x%2FGzqApu8idKWnLRBPYdeH3K2bm0TDbXuOEkU1mdpCyFIBm7%2B2pNbx0HGQlTW3tAbiHWhwVdy0p%2FEt7tkDmIe66SpStA8uPBo36AD00nlEips7jxZEjnAYF4w38DVAeL5jCQvK6xBjqxAZ4Z1X3hUyfov8klf0dNokOaYTstnfamuDerU%2BSEBtS42PFpQSNpNipzl8%2F1oad2HbBezNm60UMih2qM1gYDCnN8KAUIyVuJQMOd2Ps3GW31adoWWu420CMrCLJP%2F3riLr1WhTEbVyq5EIghsVyToqVfYg9oTj1yIUarKF1RVlBZt%2FRI8Bq1tsmKOQuocfOy9bXgQbaPODrbZGsOEcQTn8FQkajsRJ1nG7OzwOCYUndiJw%3D%3D", "publishCompletionUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@aeceba23dbb14725a3e63c3b79bded6f/handler/publish_completion", "completionEnabled": false, "autoplay": false, "completionPercentage": 0.95, "captionDataDir": null, "duration": 0.0, "saveStateUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@aeceba23dbb14725a3e63c3b79bded6f/handler/xmodule_handler/save_user_state", "ytApiUrl": "https://www.youtube.com/iframe_api", "start": 0.0, "transcriptLanguage": "en", "lmsRootURL": "https://openlearninglibrary.mit.edu", "prioritizeHls": false, "autoAdvance": false, "transcriptLanguages": {"en": "English"}, "autohideHtml5": false, "transcriptTranslationUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@aeceba23dbb14725a3e63c3b79bded6f/handler/transcript/translation/__lang__", "transcriptAvailableTranslationsUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@aeceba23dbb14725a3e63c3b79bded6f/handler/transcript/available_translations", "ytMetadataEndpoint": "", "recordedYoutubeIsAvailable": true, "sources": ["https://d2f1egay8yehza.cloudfront.net/MITHST932017-V005200_DTH.mp4", "https://d2f1egay8yehza.cloudfront.net/MITHST932017-V005200/MITHST932017-V005200.m3u8"], "showCaptions": "true", "end": 0.0}'
data-bumper-metadata='null'
data-autoadvance-enabled="False"
data-poster='null'
tabindex="-1"
>
<div class="focus_grabber first"></div>
<div class="tc-wrapper">
<div class="video-wrapper">
<span tabindex="0" class="spinner" aria-hidden="false" aria-label="Loading video player"></span>
<span tabindex="-1" class="btn-play fa fa-youtube-play fa-2x is-hidden" aria-hidden="true" aria-label="Play video"></span>
<div class="video-player-pre"></div>
<div class="video-player">
<div id="aeceba23dbb14725a3e63c3b79bded6f"></div>
<h4 class="hd hd-4 video-error is-hidden">No playable video sources found.</h4>
<h4 class="hd hd-4 video-hls-error is-hidden">
Your browser does not support this video format. Try using a different browser.
</h4>
</div>
<div class="video-player-post"></div>
<div class="closed-captions"></div>
<div class="video-controls is-hidden">
<div>
<div class="vcr"><div class="vidtime">0:00 / 0:00</div></div>
<div class="secondary-controls"></div>
</div>
</div>
</div>
</div>
<div class="focus_grabber last"></div>
</div>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@27fea0985e664ccab9f9eee43bd24042">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@27fea0985e664ccab9f9eee43bd24042" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>The Stages</h4>
<p>Health information systems are often described as socio-technical systems – their effect is based on the often complex interaction of technology, people, organizations and infrastructure. Therefore evaluation of such systems must examine the technology, the human elements, and the interactions between them. In practice this is reflected in the study methods including the division into <b>quantitative methods</b> and <b>qualitative methods</b>, with the former focusing on numeric measurement of effects, and the latter particularly focusing on the experiences and viewpoints of users.</p>
<p>It is important to address the needs of the different users and to examine systems at each stage from original requirements gathering to long-term performance. Evaluation has 5 stages. <i>Stage one</i> - problem definition includes requirements gathering and translation into design and architecture. Mistakes here will clearly impact the ability of the system to address user needs. <i>Stage two</i>, Bench Testing addresses the issue of ensuring that the software and hardware combination is stable, usable and ideally is shown to work well with authentic test data.Stages three to five are more conventionally thought of as HIS evaluation targets, but the distinctions are important here. <i>Stage three</i> is based on observing the use of systems in real environments. This is particularly important for environments with poor infrastructure as often seen in LMICs. Systems need to be <i>functional</i> and <i>used</i> regularly by staff, and key functions like reports and decision support tools need to function effectively.<i>Stage four</i> is where typical impact studies take place such as randomized controlled trials.<i>Stage five</i> represents monitoring the longer term use of the system once deployed. With the rapid changes in health systems and technology, a system that works in the initial implementation will not necessarily continue to do so longer term. In addition the roll out of a system to new sites often throws up new problems that need to be detected and addressed.</p>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@83bb05b19be143c0be523410e4381cf7">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@83bb05b19be143c0be523410e4381cf7" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Formative and Summative Evaluation</h4>
<p>A critical first stage in implementing and optimizing a HIS is ensuring that it is developed to address a real need and ensuring that its design and user interface is appropriate. The next stage is ensuring that it functions as expected. This includes stability, speed and responsiveness, usability, usage, and data quality, tested in the real environment with real users, representing stage three or four in the framework presented above. This type of study is often referred to as a <i>formative evaluation</i>. It is typically carried out by, or in partnership with, the system developers and results in improvement to the system and/or the implementation. These types of evaluations typically include quantitative and qualitative methods.</p>
<p>In contrast an evaluation study intended to assess the impact of the fully operational HIS is often referred to as a <i>summative evaluation</i>. These types of evaluation are usually primarily quantitative and based on study designs intended to assess causality – Randomized Controlled trials (RCTs) and Interrupted Time Series studies. In summative evaluations the evaluators are usually more independent of the developers. These studies are usually at Stage four and sometime stage five of the framework.</p>
</div>
</div>
<div class="vert vert-3" data-id="block-v1:MITx+HST.936x+1T2019+type@problem+block@52b9c1a14e144945ae393f5a6b3d4b07">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-init="XBlockToXModuleShim" data-has-score="True" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@problem+block@52b9c1a14e144945ae393f5a6b3d4b07" data-runtime-class="LmsRuntime" data-block-type="problem" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_52b9c1a14e144945ae393f5a6b3d4b07" class="problems-wrapper" role="group"
aria-labelledby="52b9c1a14e144945ae393f5a6b3d4b07-problem-title"
data-problem-id="block-v1:MITx+HST.936x+1T2019+type@problem+block@52b9c1a14e144945ae393f5a6b3d4b07" data-url="/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@problem+block@52b9c1a14e144945ae393f5a6b3d4b07/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="4"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="52b9c1a14e144945ae393f5a6b3d4b07-problem-title" aria-describedby="block-v1:MITx+HST.936x+1T2019+type@problem+block@52b9c1a14e144945ae393f5a6b3d4b07-problem-progress" tabindex="-1">
Unit Exercise
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.936x+1T2019+type@problem+block@52b9c1a14e144945ae393f5a6b3d4b07-problem-progress"></div>
<div class="problem">
<div>
<p>Identify which of the five stages of evaluation (1, 2, 3, 4 or 5) the following examples belongs to:</p>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><div id="formulaequationinput_52b9c1a14e144945ae393f5a6b3d4b07_2_1" class="inputtype formulaequationinput">
<div class="unanswered">
<label class="problem-group-label" for="input_52b9c1a14e144945ae393f5a6b3d4b07_2_1" id="label_52b9c1a14e144945ae393f5a6b3d4b07_2_1">Measuring the duration of patient triage during pilot implementation of a patient registration system</label>
<input type="text" name="input_52b9c1a14e144945ae393f5a6b3d4b07_2_1" id="input_52b9c1a14e144945ae393f5a6b3d4b07_2_1" data-input-id="52b9c1a14e144945ae393f5a6b3d4b07_2_1" value="" aria-describedby="status_52b9c1a14e144945ae393f5a6b3d4b07_2_1" size="20"/>
<span class="trailing_text" id="trailing_text_52b9c1a14e144945ae393f5a6b3d4b07_2_1"/>
<span class="status unanswered" id="status_52b9c1a14e144945ae393f5a6b3d4b07_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
<p id="answer_52b9c1a14e144945ae393f5a6b3d4b07_2_1" class="answer"/>
<div id="input_52b9c1a14e144945ae393f5a6b3d4b07_2_1_preview" class="equation">
\(\)
<img src="/static/images/spinner.bc34f953403f.gif" class="loading" alt="Loading"/>
</div>
</div>
<div class="script_placeholder" data-src="/static/js/capa/src/formula_equation_preview.b1967ab28c31.js"/>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 2" role="group"><div id="formulaequationinput_52b9c1a14e144945ae393f5a6b3d4b07_3_1" class="inputtype formulaequationinput">
<div class="unanswered">
<label class="problem-group-label" for="input_52b9c1a14e144945ae393f5a6b3d4b07_3_1" id="label_52b9c1a14e144945ae393f5a6b3d4b07_3_1">Evaluating whether an SMS-based prenatal visit appointment reminder reduces childbirth complications</label>
<input type="text" name="input_52b9c1a14e144945ae393f5a6b3d4b07_3_1" id="input_52b9c1a14e144945ae393f5a6b3d4b07_3_1" data-input-id="52b9c1a14e144945ae393f5a6b3d4b07_3_1" value="" aria-describedby="status_52b9c1a14e144945ae393f5a6b3d4b07_3_1" size="20"/>
<span class="trailing_text" id="trailing_text_52b9c1a14e144945ae393f5a6b3d4b07_3_1"/>
<span class="status unanswered" id="status_52b9c1a14e144945ae393f5a6b3d4b07_3_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
<p id="answer_52b9c1a14e144945ae393f5a6b3d4b07_3_1" class="answer"/>
<div id="input_52b9c1a14e144945ae393f5a6b3d4b07_3_1_preview" class="equation">
\(\)
<img src="/static/images/spinner.bc34f953403f.gif" class="loading" alt="Loading"/>
</div>
</div>
<div class="script_placeholder" data-src="/static/js/capa/src/formula_equation_preview.b1967ab28c31.js"/>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 3" role="group"><div id="formulaequationinput_52b9c1a14e144945ae393f5a6b3d4b07_4_1" class="inputtype formulaequationinput">
<div class="unanswered">
<label class="problem-group-label" for="input_52b9c1a14e144945ae393f5a6b3d4b07_4_1" id="label_52b9c1a14e144945ae393f5a6b3d4b07_4_1">Biannual reporting of adverse drug events after roll-out of a pharmacy system</label>
<input type="text" name="input_52b9c1a14e144945ae393f5a6b3d4b07_4_1" id="input_52b9c1a14e144945ae393f5a6b3d4b07_4_1" data-input-id="52b9c1a14e144945ae393f5a6b3d4b07_4_1" value="" aria-describedby="status_52b9c1a14e144945ae393f5a6b3d4b07_4_1" size="20"/>
<span class="trailing_text" id="trailing_text_52b9c1a14e144945ae393f5a6b3d4b07_4_1"/>
<span class="status unanswered" id="status_52b9c1a14e144945ae393f5a6b3d4b07_4_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
<p id="answer_52b9c1a14e144945ae393f5a6b3d4b07_4_1" class="answer"/>
<div id="input_52b9c1a14e144945ae393f5a6b3d4b07_4_1_preview" class="equation">
\(\)
<img src="/static/images/spinner.bc34f953403f.gif" class="loading" alt="Loading"/>
</div>
</div>
<div class="script_placeholder" data-src="/static/js/capa/src/formula_equation_preview.b1967ab28c31.js"/>
</div></div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 4" role="group"><div id="formulaequationinput_52b9c1a14e144945ae393f5a6b3d4b07_5_1" class="inputtype formulaequationinput">
<div class="unanswered">
<label class="problem-group-label" for="input_52b9c1a14e144945ae393f5a6b3d4b07_5_1" id="label_52b9c1a14e144945ae393f5a6b3d4b07_5_1">Survey of community health workers to elicit feedback after a demo of a mobile health app</label>
<input type="text" name="input_52b9c1a14e144945ae393f5a6b3d4b07_5_1" id="input_52b9c1a14e144945ae393f5a6b3d4b07_5_1" data-input-id="52b9c1a14e144945ae393f5a6b3d4b07_5_1" value="" aria-describedby="status_52b9c1a14e144945ae393f5a6b3d4b07_5_1" size="20"/>
<span class="trailing_text" id="trailing_text_52b9c1a14e144945ae393f5a6b3d4b07_5_1"/>
<span class="status unanswered" id="status_52b9c1a14e144945ae393f5a6b3d4b07_5_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
<p id="answer_52b9c1a14e144945ae393f5a6b3d4b07_5_1" class="answer"/>
<div id="input_52b9c1a14e144945ae393f5a6b3d4b07_5_1_preview" class="equation">
\(\)
<img src="/static/images/spinner.bc34f953403f.gif" class="loading" alt="Loading"/>
</div>
</div>
<div class="script_placeholder" data-src="/static/js/capa/src/formula_equation_preview.b1967ab28c31.js"/>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Unit Exercise" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_52b9c1a14e144945ae393f5a6b3d4b07" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_52b9c1a14e144945ae393f5a6b3d4b07">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="52b9c1a14e144945ae393f5a6b3d4b07-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="52b9c1a14e144945ae393f5a6b3d4b07-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="52b9c1a14e144945ae393f5a6b3d4b07-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="True">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-init="VerticalStudentView" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@vertical+block@85c3b1d64b8e4ea9aa80d8b56cc2d9e0" data-runtime-class="LmsRuntime" data-block-type="vertical" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<h2 class="hd hd-2 unit-title">Quantitative Evaluations</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@f968eed4121e4e8d864f7e9125c2d313">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@f968eed4121e4e8d864f7e9125c2d313" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Introduction</h4>
<p>The main focus of this section in quantitative evaluation. In these studies the goal is to assess the impact of using an HIS on some measurable end points. These could for example be the number of errors in drug prescribing or number of missed antennal clinic appointments. Quantitative evaluations tend to be the best known and most commonly reported in the literature. They can provide rigorous evidence of the impact of the system on care but may be less effective at explaining the underlying processes and socio-technical issues that explain results, a key role for qualitative research studies. There are a range of study types that include:</p>
<ul>
<li><b>Demonstration studies:</b> these assess whether a system behaves as expected based on its design</li>
<li><b>Comparative studies:</b> these include intervention and control groups to allow comparison of the intervention and usual care. These are the most common type in the literature.</li>
</ul>
<p></p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@9e9d5cf40f5b4b5eb4aa43299ba8dd0f">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@9e9d5cf40f5b4b5eb4aa43299ba8dd0f" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Control groups</h4>
<p>Scientific studies of the impact of an intervention require a comparison group that stays the same when the intervention is introduced. There are many types of controls:</p>
<ul>
<ul>
<li>Historical controls</li>
<li>Contemporary controls</li>
<li>Matched controls</li>
<li>Randomized controls</li>
<li>Time series controls</li>
<ul></ul>
</ul>
</ul>
<p>One of the simplest designs is a “before and after” study of clinical data collection and use carried out when a health information system is implemented. If there are any changes measured in process and outcomes these may be due to the new system. Unfortunately these changes may be, and often are, due to unrelated changes in the health system, staffing, other technologies etc. Such <i>historical controls</i> do not work well for studies of complex interventions particularly in LMICs where many changes may be happening at once. <i>Contemporary controls</i> are other sites, subjects or groups that are compared to the intervention group. That approach can avoid the biases due to unrelated changes in the health system seen in historical controls, but unfortunately can introduce a different set of potential biases. The other sites may be different from the intervention sites and therefore not provide a valid comparison. Matched controls are selected by statistical comparison with the intervention sites to be similar on key variables such as patient numbers, age, gender, severity of illness and characteristics like infrastructure, power, networking, and staff IT skills.</p>
</div>
</div>
<div class="vert vert-2" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@e5b02d1f719446ec83a11da4590706e9">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@e5b02d1f719446ec83a11da4590706e9" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Establishing Causality</h4>
<p>As noted earlier, when a HIS is implemented there are likely to be other changes in the health system that may be responsible for any observed differences. The best accepted way to show that changes are due to the HIS is to carry out a RCT. The key features of an RCT are the presence of one or more control groups that don’t have the system and the randomization in the selection of these groups. If control subjects or groups are selected by non-random methods it is very difficult to avoid baseline differences between the groups that can lead to bias. In typical RCTs for testing medications, there is an additional feature – double blinding. This means ensuring that neither the patient nor the doctor is aware of whether the patient is receiving the drug or the placebo. This reduces any bias in assessment of the patient. While such an approach could be valuable in studying HIS it is rarely feasible to hide the use of the HIS.</p>
<p>There are other ways in which it can be possible to attribute causality to a HIS. One example is an <b>Interrupted Time Series.</b> In this case a group is studied (e.g. a group of patients) and after a certain period (period 1) the HIS is implemented for a while (period 2), then deactivated (period 3). This may be repeated multiple times. If there are improvements in the study group that are only seen with the HIS then disappear when it is deactivated it is likely they are due to the HIS, not other unrelated changes. An example is a decision support system for chest pain patients incorporated into an ECG machine. The DSS was switched on and off multiple times (TIPI Selker H). A variant of this approach is studying the roll out of an HIS such as an EHR to many sites over a period of time in a region or country, by examining data submitted to a central site such as a national reporting system e.g. DHIS2 (see chapter 41). If site implementations are carried out on multiple known dates then any changes in submitted data that occur consistently after the HIS is implemented are likely to be due to the system.Other methods for showing causality that have been used rely on “natural experiments” when the implementation of the HIS occurs in some site but not others due to factors unrelated to the local health system.</p>
</div>
</div>
<div class="vert vert-3" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@6f507147084c41749be61992a3854b24">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@6f507147084c41749be61992a3854b24" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Utilization focused evaluation</h4>
<p>An important innovation for studies in LMICs is Utilization Focused Evaluation (UFE). This is an approach to ensuring that evaluations are of relevance and use to key stakeholders in the organization implementing the HIS, and are not just focused on questions of interest to academics and policy makers for example. UFE employs standard stages: (1) Identify key stakeholders; (2) Determine their evaluation questions; (3) Design studies to answer those questions (4) Feedback results to stakeholders. A potential advantage of the approach is that involvement of stakeholders may be perceived to bias the study. This may be addressed by involvement of external researchers in the design, implementation and analysis of the study and/or following up positive findings in larger and more independent study.</p>
</div>
</div>
<div class="vert vert-4" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@ee76f4edc58844a6a1bb4678e8b42df0">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@ee76f4edc58844a6a1bb4678e8b42df0" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Biases in evaluation studies of HIS:</h4>
<p>There are a number of potential biases in evaluation of health information systems. While it is difficult to completely eliminate biases good design can minimize their effects</p>
<ul>
<ul>
<li>Volunteer effect (+): recruiting subjects that are keen to use the system and not typical of users can lead to better outcomes than typical users.</li>
<li>Assessment bias (+): in a non-blinded study, participating clinicians may rate the intervention patients as healthier or better in some way than the controls when there is no real difference.</li>
<li>Placebo effect (+): the presence of the system may change behavior due to user expectations and lead to patients reporting better outcomes.</li>
<li>Checklist effect (+): structured recording of data on a form can improve data completeness and quality and therefore improve the processes of care being studied independent of use of the HIS.</li>
<li>Hawthorn effect (+) : The monitoring of users as part of the study makes them change behavior and potentially try harder to work with the HIS.</li>
<li>Carry-over effect (-): users that have access to decision support for some patients and not others learn from the system and improve care of control patients.</li>
<li>Allocation and recruitment bias (+): evaluators recruit users or patients that are likely to benefit from use of the system.</li>
<li>Secular trends (+): the problem with historical controls, many other changes and improvements are going on at the same time as the study, making improvements hard to attribute to the HIS.</li>
</ul>
</ul>
<p></p>
</div>
</div>
<div class="vert vert-5" data-id="block-v1:MITx+HST.936x+1T2019+type@problem+block@89106d19231a47e294e37586923f1e0e">
<div class="xblock xblock-public_view xblock-public_view-problem xmodule_display xmodule_ProblemBlock" data-init="XBlockToXModuleShim" data-has-score="True" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@problem+block@89106d19231a47e294e37586923f1e0e" data-runtime-class="LmsRuntime" data-block-type="problem" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Problem"}
</script>
<div id="problem_89106d19231a47e294e37586923f1e0e" class="problems-wrapper" role="group"
aria-labelledby="89106d19231a47e294e37586923f1e0e-problem-title"
data-problem-id="block-v1:MITx+HST.936x+1T2019+type@problem+block@89106d19231a47e294e37586923f1e0e" data-url="/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@problem+block@89106d19231a47e294e37586923f1e0e/handler/xmodule_handler"
data-problem-score="0"
data-problem-total-possible="1"
data-attempts-used="0"
data-content="
<h3 class="hd hd-3 problem-header" id="89106d19231a47e294e37586923f1e0e-problem-title" aria-describedby="block-v1:MITx+HST.936x+1T2019+type@problem+block@89106d19231a47e294e37586923f1e0e-problem-progress" tabindex="-1">
Unit Exercise
</h3>
<div class="problem-progress" id="block-v1:MITx+HST.936x+1T2019+type@problem+block@89106d19231a47e294e37586923f1e0e-problem-progress"></div>
<div class="problem">
<div>
<div class="wrapper-problem-response" tabindex="-1" aria-label="Question 1" role="group"><p>Select the type of bias described in the following example:</p>
<div class="choicegroup capa_inputtype" id="inputtype_89106d19231a47e294e37586923f1e0e_2_1">
<fieldset aria-describedby="status_89106d19231a47e294e37586923f1e0e_2_1">
<legend id="89106d19231a47e294e37586923f1e0e_2_1-legend" class="response-fieldset-legend field-group-hd">Improved dietary habits of participants in the control group who are friends with participants in the intervention group evaluating a cellphone app that provides nutritional informationthe program</legend>
<div class="field">
<input type="radio" name="input_89106d19231a47e294e37586923f1e0e_2_1" id="input_89106d19231a47e294e37586923f1e0e_2_1_choice_0" class="field-input input-radio" value="choice_0"/><label id="89106d19231a47e294e37586923f1e0e_2_1-choice_0-label" for="input_89106d19231a47e294e37586923f1e0e_2_1_choice_0" class="response-label field-label label-inline" aria-describedby="status_89106d19231a47e294e37586923f1e0e_2_1"> Checklist effect
</label>
</div>
<div class="field">
<input type="radio" name="input_89106d19231a47e294e37586923f1e0e_2_1" id="input_89106d19231a47e294e37586923f1e0e_2_1_choice_1" class="field-input input-radio" value="choice_1"/><label id="89106d19231a47e294e37586923f1e0e_2_1-choice_1-label" for="input_89106d19231a47e294e37586923f1e0e_2_1_choice_1" class="response-label field-label label-inline" aria-describedby="status_89106d19231a47e294e37586923f1e0e_2_1"> Carry-over effect
</label>
</div>
<div class="field">
<input type="radio" name="input_89106d19231a47e294e37586923f1e0e_2_1" id="input_89106d19231a47e294e37586923f1e0e_2_1_choice_2" class="field-input input-radio" value="choice_2"/><label id="89106d19231a47e294e37586923f1e0e_2_1-choice_2-label" for="input_89106d19231a47e294e37586923f1e0e_2_1_choice_2" class="response-label field-label label-inline" aria-describedby="status_89106d19231a47e294e37586923f1e0e_2_1"> Allocation/recruitment bias
</label>
</div>
<div class="field">
<input type="radio" name="input_89106d19231a47e294e37586923f1e0e_2_1" id="input_89106d19231a47e294e37586923f1e0e_2_1_choice_3" class="field-input input-radio" value="choice_3"/><label id="89106d19231a47e294e37586923f1e0e_2_1-choice_3-label" for="input_89106d19231a47e294e37586923f1e0e_2_1_choice_3" class="response-label field-label label-inline" aria-describedby="status_89106d19231a47e294e37586923f1e0e_2_1"> Secular trends
</label>
</div>
<span id="answer_89106d19231a47e294e37586923f1e0e_2_1"/>
</fieldset>
<div class="indicator-container">
<span class="status unanswered" id="status_89106d19231a47e294e37586923f1e0e_2_1" data-tooltip="Not yet answered.">
<span class="sr">unanswered</span><span class="status-icon" aria-hidden="true"/>
</span>
</div>
</div></div>
</div>
<div class="action">
<input type="hidden" name="problem_id" value="Unit Exercise" />
<div class="submit-attempt-container">
<button type="button" class="submit btn-brand" data-submitting="Submitting" data-value="Submit" data-should-enable-submit-button="True" aria-describedby="submission_feedback_89106d19231a47e294e37586923f1e0e" >
<span class="submit-label">Submit</span>
</button>
<div class="submission-feedback" id="submission_feedback_89106d19231a47e294e37586923f1e0e">
<span class="sr">Some problems have options such as save, reset, hints, or show answer. These options follow the Submit button.</span>
</div>
</div>
<div class="problem-action-buttons-wrapper">
</div>
</div>
<div class="notification warning notification-gentle-alert
is-hidden"
tabindex="-1">
<span class="icon fa fa-exclamation-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="89106d19231a47e294e37586923f1e0e-problem-title">
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification warning notification-save
is-hidden"
tabindex="-1">
<span class="icon fa fa-save" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="89106d19231a47e294e37586923f1e0e-problem-title">None
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
<div class="notification general notification-show-answer
is-hidden"
tabindex="-1">
<span class="icon fa fa-info-circle" aria-hidden="true"></span>
<span class="notification-message" aria-describedby="89106d19231a47e294e37586923f1e0e-problem-title">Answers are displayed within the problem
</span>
<div class="notification-btn-wrapper">
<button type="button" class="btn btn-default btn-small notification-btn review-btn sr">Review</button>
</div>
</div>
</div>
"
data-graded="True">
<p class="loading-spinner">
<i class="fa fa-spinner fa-pulse fa-2x fa-fw"></i>
<span class="sr">Loading…</span>
</p>
</div>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-init="VerticalStudentView" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@vertical+block@ee2fd985b6d6459e8e2c8cbcfb7ff681" data-runtime-class="LmsRuntime" data-block-type="vertical" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<h2 class="hd hd-2 unit-title">Qualitative Evaluations</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@29c86ba99fd546458920401e4853e033">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@29c86ba99fd546458920401e4853e033" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Introduction</h4>
<p>These studies focus on the experience of the users of a HIS and assess issues like user satisfaction, ease of use, barrier to effective use of HIS, and social and cultural influences on adoption and use of systems. Clearly these issues can be of high relevance in LMICs with a range of environments and limited experience with information technology. This information is valuable in assessing how users interact with the system and whether these interactions are positive or negative. A key benefit of such studies is understanding how well the HIS fits with the workflow and activities of users and how the system design can be improved. User feedback usually provides important insights into the likelihood that the system will scale and also have a beneficial impact.</p>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@55d53f63850647499c67100a4c6e8487">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@55d53f63850647499c67100a4c6e8487" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Qualitative Evaluation of HIS</h4>
<p>Kaplan and Maxwell describe the use of qualitative methods for health informatics in detail. They suggest that this approach can be helpful for the following research questions:</p>
<ul>
<ul>
<li>To determine what might be important to measure, why measured results are as they are, or if the subject of study cannot be measured easily</li>
<li>To understand not only what happened, or what people are responding to, but why; to understand how people think or feel about something and why they think that way, what their perspectives and situations are and how those influence what is happening: to understand and explore what a technology (such as an newborn nursery telemonitoring system) or practice (such as using a computer to access health information) means to people</li>
<li>To investigate the influence of social, organizational, and cultural context on the area of study, and vice versa</li>
<li>To examine causal processes, and not simply what causal relationships exist</li>
<li>To study processes as they develop and emerge, rather than in outcomes or impacts; for example, to investigate the development process for the application under study in parallel with that process so that you can improve the application development as it progresses.</li>
</ul>
</ul>
<p>Methods used in qualitative studies of HIS include semi-structured questionnaires to elicit user views and experiences, focus groups, and observation of users interacting with systems under real world or experimental conditions. Rose et al describe the use of “qualitative studies to improve the usability of an EMR” emphasizing the importance of assessing user needs and experiences with a “results manager” component of an EMR. They used Task Analysis and Focus Groups and combined the results to make recommendations for improving the design of the results manager. In another example Zakane and colleagues used semi structured interviews to study “Needs of and attitudes towards a computerized clinical decision support system in rural Burkina Faso” and found interest in and support for HIS and decisions support systems to improve maternal and child health, but concerns that the systems would be difficult to learn, and might increase work load.</p>
<p>These methods also help researchers to understanding the meaning and context of items being studied and the underlying dynamics of the processes and the way they impacts on users. Related to these ideas, Farach and colleagues use semi-structured interviews conducted mainly by “Skype” to assess the impact of HIS initiatives in addressing the health needs of underserved populations in Latin America and the Caribbean. They showed that HIS can benefit the health needs of such populations.</p>
<p>Some researchers go as far as arguing that quantitative, sometimes called “objectivist” studies cannot capture the important socio-technical interactions that are the real determinants of success and failure of HIS, and therefore are not valid without qualitative studies. There are many examples of technically sound HIS projects that failed due to sociocultural issues. Conversely there are many project that failed due to poor design, development and implementation of software or infrastructure weaknesses. As this chapter should make clear there are a range of factors that need to be considered in effective evaluation studies and qualitative research methods are a key element in the mix. While there is not space here to provide an in depth discussion of the principles and techniques of qualitative research, readers are encouraged to consult the reading list and also consider collaborating with qualitative researchers in their studies.</p>
<p></p>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-init="VerticalStudentView" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@vertical+block@dd8242276529455f8453a8ccced801d4" data-runtime-class="LmsRuntime" data-block-type="vertical" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<h2 class="hd hd-2 unit-title">Economic Evaluation</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@5539d258ca59449d844dea176bf365e8">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@5539d258ca59449d844dea176bf365e8" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Cost Studies</h4>
<p>One area of HIS deployment in LMICs that currently has very little evaluation evidence is the costing of these systems. There are several key principles that are important here in assessing the real costs of HIS and comparing those with evidence of benefits. Readers are encouraged to explore this important topic in more detail such as the following references.</p>
<p>A basic principle in costing is in accurately measuring the true cost of an HIS. This requires assessing the following items required for:</p>
<ul>
<li>Hardware and IT costs</li>
<li>Improvements needed in infrastructure such as building work, power, networking, and air conditioning</li>
<li>Software development and/or purchase including all adaption and improvement costs</li>
<li>Implementation costs including staff time for training and reduced productivity during change over</li>
<li>Training of users and support staff</li>
<li>IT support staff</li>
</ul>
<p>In addition the costs of <i>running the system over several years </i>must be measured. These include:</p>
<ul>
<li>Ongoing staff costs for IT support, training, data management, analysis etc.</li>
<li>Replacement hardware such as servers, clients and networking</li>
<li>Supplies and monthly costs such as printing, network subscriptions, power etc.</li>
<li>Ongoing software costs which could include licenses, essential improvements, upgrades and security fixes</li>
</ul>
<p>Combining these items creates the principle of <i>Total Cost of Ownership</i>. This is critical in assessing which HIS projects make most sense as the initial costs may give a misleading impression of the longer term costs. This is especially true with proprietary software with high license fees and projects dependent on reliable network connections which maybe expensive (and possibly unreliable despite the cost).</p>
<p>More advanced economic evaluations take into account the impact of the HIS as well as the cost. These include <i>Cost Effectiveness</i> analysis where the total cost of ownership is combined with measured benefits of the HIS to show the cost of a specific measured improvement per specific health outcome, patient or health facility. <i>Cost Benefit</i> analysis is similar except that it considers an array of benefits, and assigns a financial value to each – a more difficult and controversial approach especially in LMICs. Both techniques rely on an accurate evaluation of the actual benefits of the HIS so must be combined with rigorous outcome studies.</p>
<p>Two examples of costing studies illustrate some of these key principles. Saronga and colleagues examined the cost of implementing a decision support systems for maternal care in Tanzania. They detail the many items that need to be costed including vehicles and transport, computers and IT hardware, furniture, buildings, software, personnel, training, supplies and communication. These were grouped into “installation and operation costs; capital and recurrent costs; and fixed and variable costs”. The total cost reported was $185,927 USD with 77% incurred in the installation phase. Training and software were the largest cost item at 33% and 32% of total cost respectively.</p>
<p>A study by Driessen and colleagues looked at the potential cost savings from implementing an EHR system in a hospital Lilongwe, Malawi. They costed both the implementation of the EHR and the running cost over 3 - 5 years. They also modeled the costs that could be saved by improving efficiency of processes such as the labeling and tracking of laboratory samples. The hospital typically had substantial losses due to discarded samples and repeating tests, due to poor labeling. This problem was addressed by the creation of bar coded ID labels and cards from the EMR system. An actual prospective evaluation is required to assess the true costs and savings</p>
</div>
</div>
</div>
</div>
<div class="xblock xblock-public_view xblock-public_view-vertical" data-init="VerticalStudentView" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@vertical+block@a255123ac552436d8c5a73eeca7f2a8f" data-runtime-class="LmsRuntime" data-block-type="vertical" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<h2 class="hd hd-2 unit-title">Barriers to Evaluation</h2>
<div class="vert-mod">
<div class="vert vert-0" data-id="block-v1:MITx+HST.936x+1T2019+type@video+block@cfc31340f2b34704a8fe4116acce6475">
<div class="xblock xblock-public_view xblock-public_view-video xmodule_display xmodule_VideoBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@video+block@cfc31340f2b34704a8fe4116acce6475" data-runtime-class="LmsRuntime" data-block-type="video" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "Video"}
</script>
<h3 class="hd hd-2">Barriers to Evaluation</h3>
<div
id="video_cfc31340f2b34704a8fe4116acce6475"
class="video closed"
data-metadata='{"speed": null, "saveStateEnabled": false, "streams": "1.00:Ftr8wImCumE", "savedVideoPosition": 0.0, "ytTestTimeout": 1500, "generalSpeed": 1.0, "poster": "https://mitx-edx-video-meta-storage.s3.amazonaws.com/media/video-images/9689823716c641a49f05cd3387e67126.jpeg?Expires=1714136656&AWSAccessKeyId=ASIA4QDFFSURSAMMO4WA&Signature=n6sVIeKUSOMcwbSD4SLZCnK5bwQ%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEHUaCXVzLWVhc3QtMSJHMEUCIDgpsiMUMvtU%2BUkvEvgdEEjHJ8vgnN4V4h3%2F9z4USHLPAiEA3vw2OU%2F9k2EBtFnNV7NnN4CDaBu9V7uh2pez5%2FPXgL0qwwUIvv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARADGgw4NTkyMDU2Mzc0MTEiDDFbNXwLSS8FzBirSSqXBVXRa5MzUWeMZQl%2B4KenRrtNMSSax%2FGns%2Bxx4JigP58Cx212DKqpvpXjBEOXYH1g187Bx5bii9w%2Bm5btq7rahKFjFWGZE3ZESWoBZHDareII25IjkDCPmGskiLFgeB7w1W1FnpqvLdZ2iguKnVUypb8THpgei990qCPBIXyCVjfJgxgEdcR%2B4AJhz3Ac%2BKKkXyhbB9Ob7xt%2B6lBtHRSPGZ%2B2iY1b37weOXi07Oy%2BwAJrLM%2FYP26BGswnMYG2V0ynlLkIVsS4b0Ig0ehvADts4L8yAvaTBsanYkAJyl8LIhdsLIyKm3WeqeHl2rXPii17Br8xVsC6lnnbnmfiQBzmwkxRKk06LSUSD8fIzTCzwcJaYSuvuPqcn%2Fq0LL49l6NyhoqPb3zVR0yE77RXChS0%2BFRN7s3Y8SzNU69LPsAzHlZZT97tiI%2BeQdP%2Fogd9voUoF5enSEeuhz808%2F0l79SytPW8vxlawF%2FxhLJPxkHzsEBNAbWE08Olw3Xqfk2QhYtc822S51Bzy45OwzxHTOvQ3GKZebI298zzHTUJaUu37f51zPu7yjgZ4Xl3xMTFkBQS%2FSKufmdWobRnDAhNN0YiVlF2vwlCcjSzOq3hIPaTeEIuf3dvhfk4Af%2F%2FjWJ5ZikVwnY1O%2FFcwTG9qvNVLhGqFblxM5PGr3cBhHm4NAAukqhYtSukoFyRXiyst7kvNGWw4s886avaJFw%2BGRsyvdSm1j2VvgiZR0R5RfL1Sjs68VbpE0Pogft8x%2FGzqApu8idKWnLRBPYdeH3K2bm0TDbXuOEkU1mdpCyFIBm7%2B2pNbx0HGQlTW3tAbiHWhwVdy0p%2FEt7tkDmIe66SpStA8uPBo36AD00nlEips7jxZEjnAYF4w38DVAeL5jCQvK6xBjqxAZ4Z1X3hUyfov8klf0dNokOaYTstnfamuDerU%2BSEBtS42PFpQSNpNipzl8%2F1oad2HbBezNm60UMih2qM1gYDCnN8KAUIyVuJQMOd2Ps3GW31adoWWu420CMrCLJP%2F3riLr1WhTEbVyq5EIghsVyToqVfYg9oTj1yIUarKF1RVlBZt%2FRI8Bq1tsmKOQuocfOy9bXgQbaPODrbZGsOEcQTn8FQkajsRJ1nG7OzwOCYUndiJw%3D%3D", "publishCompletionUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@cfc31340f2b34704a8fe4116acce6475/handler/publish_completion", "completionEnabled": false, "autoplay": false, "completionPercentage": 0.95, "captionDataDir": null, "duration": 0.0, "saveStateUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@cfc31340f2b34704a8fe4116acce6475/handler/xmodule_handler/save_user_state", "ytApiUrl": "https://www.youtube.com/iframe_api", "start": 0.0, "transcriptLanguage": "en", "lmsRootURL": "https://openlearninglibrary.mit.edu", "prioritizeHls": false, "autoAdvance": false, "transcriptLanguages": {"en": "English"}, "autohideHtml5": false, "transcriptTranslationUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@cfc31340f2b34704a8fe4116acce6475/handler/transcript/translation/__lang__", "transcriptAvailableTranslationsUrl": "/courses/course-v1:MITx+HST.936x+1T2019/xblock/block-v1:MITx+HST.936x+1T2019+type@video+block@cfc31340f2b34704a8fe4116acce6475/handler/transcript/available_translations", "ytMetadataEndpoint": "", "recordedYoutubeIsAvailable": true, "sources": ["https://d2f1egay8yehza.cloudfront.net/MITHST932017-V004600_DTH.mp4", "https://d2f1egay8yehza.cloudfront.net/MITHST932017-V004600/MITHST932017-V004600.m3u8"], "showCaptions": "true", "end": 0.0}'
data-bumper-metadata='null'
data-autoadvance-enabled="False"
data-poster='null'
tabindex="-1"
>
<div class="focus_grabber first"></div>
<div class="tc-wrapper">
<div class="video-wrapper">
<span tabindex="0" class="spinner" aria-hidden="false" aria-label="Loading video player"></span>
<span tabindex="-1" class="btn-play fa fa-youtube-play fa-2x is-hidden" aria-hidden="true" aria-label="Play video"></span>
<div class="video-player-pre"></div>
<div class="video-player">
<div id="cfc31340f2b34704a8fe4116acce6475"></div>
<h4 class="hd hd-4 video-error is-hidden">No playable video sources found.</h4>
<h4 class="hd hd-4 video-hls-error is-hidden">
Your browser does not support this video format. Try using a different browser.
</h4>
</div>
<div class="video-player-post"></div>
<div class="closed-captions"></div>
<div class="video-controls is-hidden">
<div>
<div class="vcr"><div class="vidtime">0:00 / 0:00</div></div>
<div class="secondary-controls"></div>
</div>
</div>
</div>
</div>
<div class="focus_grabber last"></div>
</div>
</div>
</div>
<div class="vert vert-1" data-id="block-v1:MITx+HST.936x+1T2019+type@html+block@3115a71c560d412dbc7799b94abdae28">
<div class="xblock xblock-public_view xblock-public_view-html xmodule_display xmodule_HtmlBlock" data-init="XBlockToXModuleShim" data-has-score="False" data-runtime-version="1" data-graded="True" data-usage-id="block-v1:MITx+HST.936x+1T2019+type@html+block@3115a71c560d412dbc7799b94abdae28" data-runtime-class="LmsRuntime" data-block-type="html" data-request-token="c8edef5c03cc11efb64602329aca76dd" data-course-id="course-v1:MITx+HST.936x+1T2019">
<script type="json/xblock-args" class="xblock-json-init-args">
{"xmodule-type": "HTMLModule"}
</script>
<h4>Conclusions and Future Directions</h4>
<p>Evaluation studies are intimately linked to methods and processes to improve the quality of health information systems projects. Working through the different stages of evaluation and addressing the needs of key stakeholders should ensure that appropriately targeted, well designed and programmed HIS are created that are also easy to use and viewed as helpful by users. A high quality HIS of this sort will not guarantee success in all project – the complexities of different organizations and cultures may lead to unanticipated problems. However most big failures of health information systems are attributable to major errors in user requirements gathering, design, programming or implementation – known and preventable problems. Frequently systems are not complete or not designed for the purpose and workflow they are being used for, or they are difficult to use and lack user testing. A particular challenge in LMICs are HIS that require much better infrastructure than is available, including power, networking, internet and effective IT support. The process of monitoring daily usage of HIS including uptime, page views and data completeness goes a long way to ensure that failures are detected early offering opportunities to correct the problem or in some case cancel the project and go back to the drawing board.</p>
<p>Another strategy for successful system implementation is to combine “bottom up” and “top down” approaches.The bottom up approach ensures that HIS projects address clinical needs and problems, focus on local ownership and priorities, and prioritize data quality. These approaches help to ensure that the project works well in the original site. The top down approach includes planning for wide use, a horizontal approach to healthcare delivery where systems can cover a whole range of diseases and clinical problems, along with a focus on creating a core data set and supporting open standards and interoperability. Both approaches should be married along with a focus on evaluation and evidence based decision making.</p>
<p>Finding the resources, expertise and time to carry out effective evaluation of HIS remains a concern for many organizations, but as described here there are many different approaches that can be applied. In addition even small, formative evaluations can be very useful both for the implementing organization and as lessons for the wider community. The alternative cost of avoiding evaluation can be very high – large scale projects that fail in expensive and embarrassing ways due to elementary and preventable errors.</p>
</div>
</div>
</div>
</div>
© All Rights Reserved