Unlocking DataOps Success: The Crucial Role of Orchestration with Basil Faruqui at BMC

Data has long been recognized as the essential currency within enterprises, reaching right up to the executive levels. Analysts and thought leaders emphasize the critical involvement of CEOs in data initiatives. However, an important aspect often overlooked is that numerous data projects fail to make it to production. A report by Gartner in 2016 highlighted that only 15% of such projects succeed.

The effective operationalization of data projects is pivotal in transforming an overwhelming amount of data into a functional digital transformation strategy. DataOps builds on the foundation laid by DevOps, but Gartner also warns that organizations lacking a sustainable data and analytics operationalization framework by 2024 may experience delays in their initiatives of up to two years.

Good orchestration is vital for successful operationalization, as explained by Basil Faruqui, director of solutions marketing at BMC. He mentions, “When constructing a data pipeline—be it for a simple business intelligence project or a more sophisticated AI or machine learning initiative—you must consider data ingestion, storage, processing, and insights. Each of these four phases employs various underlying technologies, and consensus exists on the necessity of automation in production.”

BMC’s Control-M, particularly BMC Helix Control-M, has been an integral tool for numerous organizations for over three decades, enabling the execution of hundreds of thousands of batch jobs daily while optimizing intricate operations like supply chain management. As technological landscapes continue to evolve—from on-premises to cloud solutions—combined with rising demands for SaaS-based orchestration, the rollout of BMC Helix Control-M in 2020 was an obvious progression.

As a well-established company founded in 1980, BMC has many clients who have successfully integrated Control-M into modern use cases. A notable example is Hershey’s, a massive food manufacturer that grapples with the complexities of a time-sensitive supply chain, especially concerning perishable goods. The company has adopted cloud solutions like Azure, transitioning various existing ETL applications, while operating within a complicated SAP environment. According to Hershey’s analyst Todd Lightner, Control-M “literally runs our business.”

Faruqui elaborates on the steps of data ingestion, storage, processing, and insights, illustrating how Hershey’s would manage significant holiday campaigns or decide product shipments. “It’s all data-driven,” he notes. “They pull data from numerous systems of record and external sources, channeling it into vast data lakes. AI and ML algorithms analyze this information, which in turn feeds into an analytics layer that allows business executives to view dashboards and reports for informed decision-making.”

This highlights Hershey’s effective implementation of orchestration and automation via Control-M as a strategic advantage. However, there’s another crucial consideration. DataOps is a key element of BMC’s strategy, but it’s not the sole focus. Faruqui states, “Data pipelines depend on applications both above and below them.” Taking Hershey’s as an example, when determining promotional strategies, data may originate from SAP, a system that constantly evolves through workflows.

Faruqui raises an important question: “How does the data pipeline recognize that SAP has completed its updates, and the data is ready for processing? Once the strategy is devised, this information must be relayed back to SAP, as the ordering of raw materials will occur within ERPs, not in the data pipeline.”

Thus, the orchestration capabilities of Control-M are essential in this intricate ecosystem, ensuring seamless data workflows and integration across platforms.

Samoa, Andorra, Angola, Anguilla, Antarctica, Antigua and Barbuda, Argentina, Armenia, Aruba, Australia, Austria, Azerbaijan, Bahamas, Bahrain, Bangladesh, Barbados, Belarus, Belgium, Belize, Benin, Bermuda, Bhutan, Bolivia, Bonaire, Sint Eustatius and Saba, Bosnia and Herzegovina, Botswana, Bouvet Island, Brazil, British Indian Ocean Territory, Brunei Darussalam, Bulgaria, Burkina Faso, Burundi, Cambodia, Cameroon, Canada, Cape Verde, Cayman Islands, Central African Republic, Chad, Chile, China, Christmas Island, Cocos Islands, Colombia, Comoros, Congo, Congo (Democratic Republic of the), Cook Islands, Costa Rica, Croatia, Cuba, Curaçao, Cyprus, Czechia, Côte d’Ivoire, Denmark, Djibouti, Dominica, Dominican Republic, Ecuador, Egypt, El Salvador, Equatorial Guinea, Eritrea, Estonia, Eswatini, Ethiopia, Falkland Islands, Faroe Islands, Fiji, Finland, France, French Guiana, French Polynesia, French Southern Territories, Gabon, Gambia, Georgia, Germany, Ghana, Gibraltar, Greece, Greenland, Grenada, Guadeloupe, Guam, Guatemala, Guernsey, Guinea, Guinea-Bissau, Guyana, Haiti, Heard Island and McDonald Islands, Holy See, Honduras, Hong Kong, Hungary, Iceland, India, Indonesia, Iran, Iraq, Ireland, Isle of Man, Israel, Italy, Jamaica, Japan, Jersey, Jordan, Kazakhstan, Kenya, Kiribati, Korea (Democratic People’s Republic of), Korea (Republic of), Kuwait, Kyrgyzstan, Lao People’s Democratic Republic, Latvia, Lebanon, Lesotho, Liberia, Libya, Liechtenstein, Lithuania, Luxembourg, Macao, Madagascar, Malawi, Malaysia, Maldives, Mali, Malta, Marshall Islands, Martinique, Mauritania, Mauritius, Mayotte, Mexico, Micronesia, Moldova, Monaco, Mongolia, Montenegro, Montserrat, Morocco, Mozambique, Myanmar, Namibia, Nauru, Nepal, Netherlands, New Caledonia, New Zealand, Nicaragua, Niger, Nigeria, Niue, Norfolk Island, North Macedonia, Northern Mariana Islands, Norway, Oman, Pakistan, Palau, Palestine (State of), Panama, Papua New Guinea, Paraguay, Peru, Philippines, Pitcairn, Poland, Portugal, Puerto Rico, Qatar, Romania, Russian Federation, Rwanda, Réunion, Saint Barthélemy, Saint Helena (Ascension and Tristan da Cunha), Saint Kitts and Nevis, Saint Lucia, Saint Martin, Saint Pierre and Miquelon, Saint Vincent and the Grenadines, Samoa, San Marino, Sao Tome and Principe, Saudi Arabia, Senegal, Serbia, Seychelles, Sierra Leone, Singapore, Sint Maarten, Slovakia, Slovenia, Solomon Islands, Somalia, South Africa, South Georgia and the South Sandwich Islands, South Sudan, Spain, Sri Lanka, Sudan, Suriname, Svalbard and Jan Mayen, Sweden, Switzerland, Syria (Arab Republic), Taiwan, Tajikistan, Tanzania (United Republic of), Thailand, Timor-Leste, Togo, Tokelau, Tonga, Trinidad and Tobago, Tunisia, Turkmenistan, Turks and Caicos Islands, Tuvalu, Türkiye, US Minor Outlying Islands, Uganda, Ukraine, United Arab Emirates, United Kingdom, United States, Uruguay, Uzbekistan, Vanuatu, Venezuela, Viet Nam, Virgin Islands (British), Virgin Islands (U.S.), Wallis and Futuna, Western Sahara, Yemen, Zambia, Zimbabwe, Åland Islands.