{"id":1451,"date":"2022-07-19T15:38:35","date_gmt":"2022-07-19T15:38:35","guid":{"rendered":"https:\/\/www.clouddatainsights.com\/?p=1451"},"modified":"2022-07-19T15:38:38","modified_gmt":"2022-07-19T15:38:38","slug":"sound-data-management-practices-can-pay-off-now-and-later-heres-how","status":"publish","type":"post","link":"https:\/\/www.clouddatainsights.com\/sound-data-management-practices-can-pay-off-now-and-later-heres-how\/","title":{"rendered":"Sound Data Management Practices Can Pay Off Now and Later. Here\u2019s How.\u00a0\u00a0"},"content":{"rendered":"<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S.jpg\" alt=\"\" class=\"wp-image-1452\" width=\"500\" height=\"334\"\/><figcaption>Taking a proactive and strategic approach to data management can save time, money, and resources while unlocking even more powerful insights.<\/figcaption><\/figure><\/div>\n\n\n<p>Modern companies know that analytics needs to be at the heart of their decision-making and business strategy. That\u2019s why many have invested in pathways to access vast amounts of data. But now, they are encountering a new challenge: <a href=\"https:\/\/www.datacenterdynamics.com\/en\/opinions\/saving-money-smarter-storage-why-data-management-becoming-business-critical\/\" target=\"_blank\" rel=\"noreferrer noopener\">Over 70% of the data generated today<\/a> is no longer structured, easy to manage, find or analyze. This challenge begs the question: What can be done to make this data meaningful to the business?<\/p>\n\n\n\n<p>Unfortunately, a \u201cone size fits all\u201d approach to data architecture and management doesn\u2019t exist. But here are different approaches that have historically been used while also introducing the lesser-known, emerging model of data mesh.<\/p>\n\n\n\n<p><strong>See also: <\/strong><a href=\"https:\/\/www.clouddatainsights.com\/enabling-innovation-with-the-right-cloud-data-architecture\/\" target=\"_blank\" rel=\"noreferrer noopener\">Enabling Innovation with the Right Cloud Data Architecture&nbsp;<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Breaking Down the Role of Traditional Data Architecture<\/strong><\/h3>\n\n\n\n<p><strong>Data Lakes<\/strong>: A <a href=\"https:\/\/www.techtarget.com\/searchdatamanagement\/definition\/data-lake#:~:text=A%20data%20lake%20is%20a,in%20files%20or%20object%20storage.\">data<\/a><a href=\"https:\/\/www.techtarget.com\/searchdatamanagement\/definition\/data-lake#:~:text=A%20data%20lake%20is%20a,in%20files%20or%20object%20storage.\" target=\"_blank\" rel=\"noreferrer noopener\"> <\/a><a href=\"https:\/\/www.techtarget.com\/searchdatamanagement\/definition\/data-lake#:~:text=A%20data%20lake%20is%20a,in%20files%20or%20object%20storage.\">lake<\/a> is a giant, central storage repository that holds a vast amount of raw data in its native format until it is needed. Data lakes help data scientists and analysts who are tasked with determining whether raw data from an organization\u2019s datasets can be turned into actionable insights. Through its flat architecture, the data lake provides more flexibility, storage, and usage at a lower cost, so that the data from the business systems can be replicated into a single repository.<\/p>\n\n\n\n<p>Data lakes can be beneficial for industries like oil and gas that accumulate large, complex data sets. On average, an oil company generates 1.5 terabytes of Internet of Things data daily. By leveraging data lakes for exploration, this industry can optimize directional drilling, lower operating expenses, improve safety, and stay compliant with regulatory requirements.<\/p>\n\n\n\n<p><strong>Data Warehouses<\/strong>: Once data scientists or analysts find value in the various datasets within these data lakes, that refined data and intelligence can be brought into a <a href=\"https:\/\/www.gartner.com\/en\/information-technology\/glossary\/data-warehouse\" target=\"_blank\" rel=\"noreferrer noopener\">data warehouse<\/a>. A data warehouse is a structured storage architecture used to hold cleansed and transformed data from various sources for historical reporting and large-scale decision support.<\/p>\n\n\n\n<p>Data warehouses tend to be quite large and central to business success. They require significant engineering and operational effort to build and maintain. They can be platformed in on-premises systems, cloud deployments, or data-warehouse-as-a-service offerings.<\/p>\n\n\n\n<p>The refined data and intelligence housed in a data warehouse are commonly aggregated and shaped to be more &#8220;business-friendly&#8221; to inform better reuse and decision-making for enterprises. This approach can be helpful for organizations that need to make repeatable business decisions and drive operational efficiencies.<\/p>\n\n\n\n<p>For instance, Walmart used its data warehouses to test inventory management methods in its U.S. and Canadian stores. As a result, Walmart could make more informed decisions to <a href=\"https:\/\/corporate.walmart.com\/newsroom\/2010\/02\/23\/walmart-canada-to-open-35-to-40-supercentres-in-2010#:~:text=Walmart%20Canada%20to%20Open%2035%20to%2040%20Supercentres%20in%202010\" target=\"_blank\" rel=\"noreferrer noopener\">open new locations in Canada<\/a> and <a href=\"https:\/\/www.usatoday.com\/story\/money\/business\/2016\/01\/15\/list-of-walmart-stores-closing\/78852898\/\" target=\"_blank\" rel=\"noreferrer noopener\">close certain U.S. stores<\/a> in an effort to accommodate its customers\u2019 needs.<\/p>\n\n\n\n<p><strong>Operational Data Store (ODS):<\/strong> Because the data warehouse is massive and has many moving parts, it can be difficult to update that data frequently to support fast-moving decisions.<\/p>\n\n\n\n<p>An <a href=\"https:\/\/www.techtarget.com\/searchoracle\/definition\/operational-data-store\" target=\"_blank\" rel=\"noreferrer noopener\">ODS<\/a> integrates and transforms the minimal cross-system data required to provide real-time decision support. This separates the high-compute transformations for fast intelligence from the large, regular needs of the data warehouse. Data, decisions, and alerts from the ODS are often moved into the data warehouse or data lake for archival use.&nbsp;<\/p>\n\n\n\n<p>Any organization managing minute-to-minute decisioning \u2014 patient care, manufacturing lines, or energy management are great examples \u2014 could benefit from an ODS approach.<\/p>\n\n\n\n<p><strong>See also: <\/strong><a href=\"https:\/\/www.clouddatainsights.com\/how-the-data-lakehouse-might-usurp-the-warehouse-and-the-lake\/\" target=\"_blank\" rel=\"noreferrer noopener\">How the Data \u2018Lakehouse\u2019 Might Usurp the Warehouse and the Lake<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>How Data Mesh Fits<\/strong><\/h3>\n\n\n\n<p>Alongside these three classic modes of data storage, <a href=\"https:\/\/www.datanami.com\/2022\/01\/21\/data-meshes-set-to-spread-in-2022\/\" target=\"_blank\" rel=\"noreferrer noopener\">the data mesh concept<\/a> is a growing architectural design principle that allows data scientists and analysts to examine data anywhere in a system: across the data lake, data warehouse, and ODS, as well as source systems. This creates a \u201cvirtual data hub\u201d that enables robust, enterprise-wide exploration without the cost and overhead of replicating data of unknown value into the data lake.<\/p>\n\n\n\n<p>Once value is discovered, engineering effort is used to pipe the data and intelligence into the data lake, data warehouse, and\/or operational data store for consumption.<\/p>\n\n\n\n<p>Numerous industries can benefit from data mesh. Many financial services companies are grappling with how to modernize outdated technology. The systems they typically use have been in place for 50+ years, and any attempt at updating slows down system processes and incurs risk.<\/p>\n\n\n\n<p>Healthcare is another industry that greatly benefits from the data mesh concept. It allows providers to navigate security positions tailored to HIPAA while also using patient data to improve the care experience and overall outcomes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Key Considerations when Migrating to Data Management Models<\/strong><\/h3>\n\n\n\n<p>While the value proposition for any of these data management models is compelling, it\u2019s important to recognize that organizations can face several challenges with migration.<\/p>\n\n\n\n<p>There\u2019s a sizable investment for these infrastructures (think large on-prem appliances like Netezza). Additionally, the amount of labor and skill required to maintain a system is very different compared to what it takes to build it, so organizations must adapt accordingly.<\/p>\n\n\n\n<p>In order to lay the foundation for well-governed data management, companies must:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Understand the long-term strategy<\/strong>: If a company knows what long-term analytics success looks like, it\u2019s possible to bring data into the decision-making process and provide data scientists and analysts with the tools they need to be successful.<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Be nimble and flexible<\/strong>: Ideally, organizations should put flexibility front and center when migrating to a data management solution. This way, the data architect\u2019s designs will meet the organization\u2019s needs today and grow with it based on future needs.<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Invest in the right people and skillsets<\/strong>: Because these data architectures are relatively new, there is often a knowledge gap. While the data warehouse has been around for 50+ years, newer systems like data lake or data mesh are often less understood. Business and technology leaders must be in lockstep when it comes to making critical investments and having the right expertise in place to drive these solutions. In today\u2019s talent-constrained environment, often bringing in a third-party solution provider can make sense for both implementation and day-to-day management.<\/li><\/ul>\n\n\n\n<p>There&#8217;s a business case to be made for investing in and implementing a well-governed, measured approach to data management<a>. Taking a proactive and strategic approach to data management <\/a>can save time, money, and resources while unlocking even more powerful insights that lead to business outcomes.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Taking a proactive and strategic approach to data management can save time, money, and resources while unlocking even more powerful insights that lead to business outcomes. <\/p>\n","protected":false},"author":48,"featured_media":1452,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"om_disable_all_campaigns":false,"_uag_custom_page_level_css":"","neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","neve_meta_reading_time":"","_themeisle_gutenberg_block_has_review":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[17],"tags":[62,72,63],"class_list":["post-1451","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-governance","tag-data-lake","tag-data-management","tag-data-warehouse"],"acf":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590.jpg","uagb_featured_image_src":{"full":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590.jpg",300,200,false],"thumbnail":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590-150x150.jpg",150,150,true],"medium":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-300x200.jpg",300,200,true],"medium_large":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-768x512.jpg",768,512,true],"large":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590.jpg",300,200,false],"1536x1536":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590.jpg",300,200,false],"2048x2048":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590.jpg",300,200,false],"alm-thumbnail":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590-150x150.jpg",150,150,true],"neve-blog":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-930x620.jpg",930,620,true],"rpwe-thumbnail":["https:\/\/www.clouddatainsights.com\/wp-content\/uploads\/2022\/07\/data-management-Depositphotos_63896367_S-e1658244825590-45x45.jpg",45,45,true]},"uagb_author_info":{"display_name":"Ken Seier","author_link":"https:\/\/www.clouddatainsights.com\/author\/ken-seier\/"},"uagb_comment_info":1,"uagb_excerpt":"Taking a proactive and strategic approach to data management can save time, money, and resources while unlocking even more powerful insights that lead to business outcomes.","_links":{"self":[{"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/posts\/1451"}],"collection":[{"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/users\/48"}],"replies":[{"embeddable":true,"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/comments?post=1451"}],"version-history":[{"count":4,"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/posts\/1451\/revisions"}],"predecessor-version":[{"id":1456,"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/posts\/1451\/revisions\/1456"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/media\/1452"}],"wp:attachment":[{"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/media?parent=1451"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/categories?post=1451"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.clouddatainsights.com\/wp-json\/wp\/v2\/tags?post=1451"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}