Signs of depression can be difficult to spot, even for atrained professional. But could youteach an AI to reliably pick up on subtle changes in mood and behavior?

CompanionMX, a Boston-based startup, has created a mental health tool to do exactly that. Its platform uses AI to help users suffering from depression track changes in their emotional state over time.

To do so, co-founder Skyler Place and his team had to overcome one of emotion AIs biggest hurdles building anaccurate dataset for training. Because at the end of the day, if you collect the wrong data or get sloppy with the labeling, the resulting algorithms canquickly cause more harm than good.

The trend of emotion AI has been met with skepticismby many, for exactly that reason. In some cases, these concerns tie back to the limitations of relying onfacial recognition, which some experts say results in a much-too-simplistic understanding of human emotions.

Holding PhDs in psychology and cognitive science, Place knows as well as anyone that a simplistic approach wont do.

To avoid bias and spurious correlations, CompanionMX rooted its analysis inknown symptoms of depression. Through the app, the user records a 30-second audio diary commenting on their day, which an algorithm uses to analyze their vocal patterns to determine their energy level. Meanwhile, the app collects data on their texting frequency and phone calls out, as well as distance traveled via GPS proxies for real-world behaviors like social engagement and physical activity.

Combined, these data pointsform a well-rounded picture of a patients mental state while minimizing the barrier to participation for users, Place said. Heres what his team learned from translating those data points into a reliable model.

Seven years ago, Place laid the foundation for how his team would collect emotional data.

At the time, CompanionMX and its parent company, Cogito, worked in partnership with DARPA to develop an algorithm that could track mental illness changes in veterans with PTSD. While the easiest and most affordable route to collect that data would be to conduct a self-report study, people tend to be unreliable in reporting on their own emotions, Place said.

How a person feels can change depending on the external factors around them that have nothing to do with their overall emotional state. For instance, the user might report higher levels of stress if theyre filling out a questionnaire while stuck in a long line at Starbucks. Moreover, if a person suffers from depression or trauma, they may be reluctant to talk about it.

If your source of truth is inaccurate or fuzzy, like a self-report, it doesnt matter how good your input data is, Place said. Its almost like youve created a situation where its impossible to win, and thats been a limitation in the field.

Instead, CompanionMX opted to conduct a clinical study. While Place said doing so required both time and money, it allowed them to gather a more accurate picture of their users emotions that psychologists would trust.

For the study, the team recruited 73 participants that represented a mix of gender, race and veteran and civilian status, who reported at least one symptom of PTSD or depression verified by medical professionals. They started with the theory that behavioral patterns could be identified in real time through vocal recordings and digital trace data logged on a smartphone.

That gave us a very robust, one-of-a-kind dataset to build some of the most accurate and predictive models of behaviors that relate to mental illness.

Each participant received a new phone, conducted a baseline questionnaire and had their clinical symptoms measured through a validated clinical interview with a trained clinician. For the next 12 weeks, participants recorded audio logs of how they were doing and had their phone data collected. The CompanionMX team tracked anything it felt could be linked to a users behavior state, including how many calls or texts went in and out, how the phone was handled using the gyroscope, the location of the device, screen time and battery status.

At the end, the participants went through a Structured Clinical Interview for Mental Disorders with a clinical psychologist, which is a standard method used in psychology to assess a persons mental health. By doing the study this way, the team received more than 50 million data points from a variety of inputs to go along with 20 different diagnoses and sub-symptoms of depression.

That gave us a very robust, one-of-a-kind dataset to build some of the most accurate and predictive models of behaviors that relate to mental illness, Place said.

How does phone usage change with depression? Thats one of the questions CompanionMX needed to answer to make sense of its data.

To sort through its trove of information, the team used a hypothesis-driven process called feature engineering, Place said. Based onwhat is known in psychology about depression, the team created theories for how typical symptoms would manifest through phone usage patterns. Important symptoms included depressed mood most of the day, diminished interest in activities, fatigue, and avoidance of activities, places and people.

For diminished interest, the team predicted that a person would send fewer text messages and travel less from their home. To measure fatigue, they hypothesized that a person would make fewer calls and send fewer texts.

Its a process where we try different features, evaluate their predictive power and eliminate the features that were not predictive.

In total, the team started with 14 of these input features based on the digital trace data. From there, CompanionMX began the trial-and-error process of building its algorithm models.

Thats where the model building runs through an evaluation and an iterative process, Place said. Its a process where we try different features, evaluate their predictive power and eliminate the features that were not predictive.

The initial approach predicted symptoms using all 12 weeks of data, but the results revealed that it would be more accurate to focus on data one week prior to the users clinical assessment, according to the report.

Through extensive data exploration, the team also identified additional features that didnt accurately represent user behavior. For example, the team eliminated any information regarding battery life and screen time because each device measured the information differently.

When youre designing your dataset, you need to ask yourself where the variability is coming from, Place said. Is the variability coming from human behavior, which youre trying to predict, or is it variability in some other attribute you cant control for? Screen off and on seem like simple data points, but it was actually being measured differently by different device manufacturers. So the same human behavior would lead to different outcomes.

Special DeliveryMITs Delivery AI Can Find Your Door Without a Map

One of the final rules Place had for his team when it came time to build their algorithm was that it needed to use features that were based on patterns humans can easily recognize.

When youre working with millions of data points, it can be easy to create a model that is statistically accurate but impossible to explain. Since CompanionMX aimed to be a resource for psychologists and psychiatrists, Place said it was crucial that their model made sense to them.

One lesson we learned was that you dont need to jump to the most complicated inputs or throw the whole kitchen sink into it at the start, Place said. Working from data that is as clean as possible and that you understand can be beneficial to modeling outcomes.

The team landed on three digital trace features all easy to understand in relation to human behavior: text message count, distance traveledand phone calls out.

Working from data that is as clean as possible and that you understand can be beneficial to modeling outcomes.

What success looked like for us was not just having an accurate model, but having a model that we could explain to a psychologist in a way that they would feel comfortable and confident in using to make decisions about treating someone for mental illness, Place said.

To train its second algorithm, CompanionMX needed to label 847 audio diary clips of at least 30 seconds in length.

Within those recordings, the team focused on the rate at which a person talked and their energy level.

While the speaking rate could be quantified through a process that measures the number of syllables used per minute, interpreting energy levels required a team of data labelers. In order to reduce the risk of bias, CompanionMX compiled a mix of 10 data scientists, both men and women, with backgrounds in psychology and engineering, to label the data.

It gave us the data necessary to make corrections and adjustments to account for the variability in human perception and human performance.

The labelers listened to the vocal recordings, and ranked pitch variation and vocal effort on the Likert scale of one to five. Since the data required human interpretation, the scale helped labelers find high levels of inter-rater reliability among the ratings.

It gave us the data necessary to make corrections and adjustments to account for the variability in human perception and human performance, Place said.

With the data labeled, the team built an algorithm that could conclude from the audio diary the users mean pitch variance and mean vocal effort. That information is then used in the teams final model to identify whether the person exhibits a depressed mood most of the day.

With the final algorithms in place, CompanionMX measured its models accuracy through a process of 10-fold cross-validation area under the curve testing a commonly used method to measure an algorithms true positive rate.

The model cleared CompanionMXs 50-percent threshold, accurately identifying each of the four depression symptoms at a rate higher than 50 percent of the time. But Place didnt want to stop there. In order to earn the trust of clinical professionals, the team aims to prove its usefulness through a clinical trial.

Its going to be a tool thats going to allow your doctor and the patients to be able to make better, more informed decisions.

The results of the clinical trial havent been published yet, but the process involved testing its product with a group of doctors and patients against a control group. The team is also working on expanding CompanionMX to identify other mental health conditions, like bipolar disorder.

When it comes to understanding how you really feel, however, Place knows the only people equipped to answer that question are you and your doctor. CompanionMX is just there to provide another data point.

I dont think well be in a world where AI is going to replace your doctor, Place said. Its going to be a tool thats going to allow your doctor and the patients to be able to make better, more informed decisions.

Striking the Right ToneHow Grammarly's Tone Detector Uses AI to Analyze Your Written Communication

Follow this link:
How CompanionMX Trained Its AI to Spot Signs of Depression - Built In