All About Data and Decision Making:  BIIA found this blog on by Databricks, which offers insights in how a relatively young business helps companies to manage data and to make intelligent decisions.

Financial Institutions today are still struggling to keep up with the emerging risks and threats facing their business. Managing risk, especially within the banking sector, has increased in complexity over the past several years.

First, new frameworks (such as FRTB) are being introduced that potentially require tremendous computing power and an ability to analyze years of historical data. Second, regulators are demanding more transparency and explainability from the banks they oversee. Finally, the introduction of new technologies and business models means that the need for sound risk governance is at an all-time high. However, the ability for the banking industry to effectively meet these demands has not been an easy undertaking.

Agile approach to risk management

Traditional banks relying on on-premises infrastructure can no longer effectively manage risk. Banks must abandon the computational inefficiencies of legacy technologies and build an agile Modern Risk Management practice capable of rapidly responding to market and economic volatility using data and advanced analytics.

Our work with clients shows that as new threats, such as the last decade’s financial crisis, emerge, historical data and aggregated risk models lose their predictive values quickly. Luckily, modernization is made possible today based on open-source technologies powered by cloud-native big data infrastructure that bring an agile and forward-looking approach to financial risk analysis and management.

Traditional datasets limit transparency and reliability

Risk analysts must augment traditional data with alternative datasets to explore new ways of identifying and quantifying the risk factors facing their business, both at scale and in real time. Risk management teams must be able to efficiently scale their simulations from tens of thousands up to millions by leveraging both the flexibility of cloud compute and the robustness of open-source computing frameworks like Apache SparkTM.

They must accelerate model development lifecycle by bringing together both the transparency of their experiment and the reliability in their data, bridging the gap between science and engineering and enabling banks to have a more robust approach to risk management.

Data organization is critical to understanding and mitigating risk

How data is organized and collected is critical to creating highly reliable, flexible and accurate data models. This is particularly important when it comes to creating financial risk models for areas such as wealth management and investment banking.

In the financial world, risk management is the process of identification, analysis and acceptance or mitigation of uncertainty in investment decisions.

When data is organized and designed to flow within an independent pipeline, separate from massive dependencies and sequential tools, the time to run financial risk models is significantly reduced. Data is more flexible, easier to slice and dice, so institutions can apply their risk portfolio at a global and regional level as well as firmwide.

Plagued by the limitations of on-premises infrastructure and legacy technologies, banks particularly have not had the tools until recently to effectively build a modern risk management practice. A modern risk management framework enables intraday views, aggregations on demand and an ability to future proof/scale risk assessment and management.

Replace historical returns with highly accurate predictive models

Financial risk modeling should include multiple data sources to create more predictive financial and credit risk models. A modern risk and portfolio management practice should not be solely based on historical returns but also must embrace the variety of information available today.

For example, a white paper from Atkins et al describes how financial news can be used to predict stock market volatility better than close price. As indicated in the white paper, the use of alternative data can dramatically augment the intelligence for risk analysts to have a more descriptive lens of modern economy, enabling them to better understand and react to exogenous shocks in real time.

A modern risk management model in the cloud

Avanade and Databricks have demonstrated how Apache Spark, Delta Lake and MLflow can be used in the real world to organize and rapidly deploy data into a value-at-risk (VAR) data model. This enables financial institutions to modernize their risk management practices into the cloud and adopt a unified approach to data analytics with Databricks.

Using the flexibility and scale of cloud compute and the level of interactivity in an organization’s data, clients can better understand the risks facing their business and quickly develop accurate financial market risk calculations. With Avanade and Databricks, businesses can identify how much risk can be decreased and then accurately pinpoint where and how they can quickly apply risk measures to reduce their exposure.

This blog was collaboratively written with Databricks partner Avanade. A special thanks to Dael Williamson, Avanade CTO, for his contributions.

About Databricks:

With origins in both academia and the open-source community, Databricks has always been devoted to simplifying data, sharing knowledge and pursuing truths. Founded in 2013 by the original creators of Apache Spark™, Delta Lake and MLflow, Databricks brings together data engineering, science and analytics on an open, unified platform so data teams can collaborate and innovate faster.

More than five thousand organizations worldwide —including Shell, Conde Nast and Regeneron — rely on Databricks as a unified platform for massive-scale data engineering, collaborative data science, full-lifecycle machine learning and business analytics. See real world examples.

Venture-backed and headquartered in San Francisco (with offices on four continents) and hundreds of global partners, including Microsoft, Amazon, Tableau, Informatica, Cap Gemini and Booz Allen Hamilton, Databricks is on a mission to help data teams solve the world’s toughest problems.    To learn more, follow Databricks on Twitter, LinkedIn and Facebook.