Grier Phillips (Manager, Data Science) created this brief newsletter to showcase the progress data science has made. If you have questions or suggestions, please email gphillips@emoneyadvisor.com.
AI has become wildly popular this year, spurred by the release of ChatGPT, and we are starting to see more and more applications in business and everyday life. From travel planners to advances in pharmaceutical research, AI is everywhere.
In the financial planning sector, Morgan Stanley has partnered with OpenAI to develop a conversational tool to help advisors derive insights from internal reports and data. While this is by no means an automated planning tool, its intent is to assist advisors in developing plans by hastening the research process.
Inherent in this system is trust. Morgan Stanley trusts that the tool will reliably and with fidelity derive accurate insights from their body of knowledge. As we have learned with ChatGPT this year, that trust may not always be well founded. Instances of “hallucination” are abundant, with ChatGPT providing factually inaccurate responses and even continuing to argue in their favor.
As a simple experiment, I set out to have a conversation with ChatGPT to inquire about its efficacy at solving different types of mathematical problems. The results were expected, yet insightful. ChatGPT can correctly answer computational questions, but only with small numbers. Large numbers or increasing accurate decimals present problems for ChatGPT as it is truly incapable of executing the computations, but rather repeating facts that it has previously learned. ChatGPT knows 2 + 2 = 4, as that is prevalent across the internet, but it doesn’t know the square root of 987,654,321, so it provides its best guess instead.
So ChatGPT is untrustworthy for computation. Despite this, it does understand the process to derive an answer. When presented with questions on statistics, calculus, or linear algebra, ChatGPT reproduced a set of instructions for how to solve those problems. For example, ChatGPT can reproduce instructions for how to calculate a derivative or integral. It can produce instructions on how to solve a quadratic equation or find eigenvalues. Where ChatGPT fails, is in solving specific problems themselves.
The full conversation is linked here, but the gist is that we can’t and shouldn’t trust large language models to do everything. They were developed specifically to mimic human composition and conversation augmented by a subject matter context greater than any human could ever achieve and we should remember that.
As for eMoney’s use of “AI”, we don’t currently have any plans to partner with OpenAI. However, the Data Science team continues to pair with the Financial Planning Group and the broader organization to identify opportunities to apply machine learning in the planning experience.
In other news, the Data Science team is proud to announce that Transaction Categorization has been integrated with Nexus and is currently undergoing testing before a full deployment. On the data science side, Hung Luu has implemented model monitoring so that we will, in real time, be able to gain insights into the performance of the model and the transaction data coming in. This will allow us to ensure that the predictions we send to our customers are as accurate as possible.
Spring is in full gear, and we are finally in a state where we can begin to dig into our data resources and start providing data insights. The Databricks platform has been onboarded and EMA data (big thanks to Steve Goldman, Agnel Jeyaraj, and the Analytics Team) is starting to land in our environment!
We have begun the process of building out data pipelines and generating dashboards to expose insights from this data. There have been a few hiccups along the way as we identify data discrepancies and anomalies, but we are getting closer to what we consider “gold” quality data. In addition, we are continuing to grow our data assets and will soon have access to sales/marketing and event tracking data.
Below is one of the insights Blaine Bursey has built using the EMA advisor login data, showing average number of advisor logins per quarter grouped by total client net worth.
In addition to analytics, we are continuing to work on integrating the Transaction Categorization project into Nexus and with Finance and Financial Planning on opportunities to apply machine learning to churn and recommendations for financial planning.
Hung Luu has been driving the integration with Nexus for the data science team by pushing forward our MLOps capacity in Databricks for model serving, monitoring, and retraining. Current scheduling between Aggregation and Data Science has a release date of mid to late summer.
Overall, we are well into creating an environment where we can integrate data from across the company and deliver both analytical insights and machine learning solutions to advanced business problems. Look for monthly updates moving forward!
Nice work, Data Science Team!
Good work ! Thanks for the update !