medRxiv preprint doi: https://doi.org/10.1101/19010413.

The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Prediction of Suicidal Ideation in the ​Canadian Community Health Survey - Mental Health
Component​ Using Deep Learning

1
Sneha Desai*​ab​, Myriam Tanguay-Sela*​bc​, David Benrimoh​bde , Robert Fratila​b​, Eleanor Brown​bf​,
Kelly Perlman​bg​, Ann John​h​, Marcos DelPozo-Banos​h​, Nancy Low​d​, Sonia Israel​b​, Lisa Palladini​de​,
Gustavo Turecki​dg

a) Department of Computer Science, University of Toronto, Toronto, Canada

b) Aifred Health Inc., Montreal, Canada
c) Montreal Neurological Institute, McGill University, Montreal, Canada
d) Department of Psychiatry, McGill University, Montreal, Canada
e) Faculty of Medicine, McGill University, Montreal, Canada
f) Department of Psychological and Brain Sciences, Boston University, Boston, USA
g) Douglas Mental Health University Institute, Montreal, Canada
h) Swansea University, Swansea, Wales, UK

*These authors contributed equally to this work.

Abstract

Introduction: Suicidal ideation (SI) is prevalent in the general population, and is a prominent risk factor
for suicide. However, predicting which patients are likely to have SI remains a challenge. Deep
Learning (DL) may be a useful tool in this context, as it can be used to find patterns in complex,
heterogeneous, and incomplete psychiatric datasets. An automated screening system for SI could
help prompt clinicians to be more attentive to patients at risk for suicide.

Methods: Using the Canadian Community Health Survey - Mental Health Component, we trained a DL
model based on 23,859 survey responses to predict lifetime SI on an individual patient basis. Models
were created to predict both lifetime and last 12 month SI. We reduced 582 possible model
parameters captured by the survey to 96 and 21 feature versions of the models. Models were trained
using an undersampling procedure that balanced the training set between SI and non-SI respondents;
validation was done on held-out data.

Results: AUC was used as the main model metric. For lifetime SI, the 96 feature model had an AUC
of 0.79 and the 21 feature model had an AUC of 0.75. For SI in the last 12 months the 96 feature
model had an AUC of 0.76 and the 21 feature model had an AUC of 0.69. DL outperformed random
forest classifiers.

Discussion: Although requiring further study to ensure clinical relevance and sample generalizability,
this study is a proof-of-concept for the use of DL to improve prediction of SI. This kind of model would
help start conversations with patients which could lead to improved care and, it is hoped, a reduction
in suicidal behavior.

1 Corresponding author:
David Benrimoh
david.benrimoh@mail.mcgill.ca
Phone: 514-463-7813
1025 Pine Ave W, Montreal, Quebec, H3A 1A1

1
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

1. Introduction

Suicide is one of the leading causes of death across the world, accounting for approximately
800,000 deaths each year with the number of attempts an order of magnitude higher ​(World Health
Organization [WHO], 2018)​. Globally, suicide accounts for 16% of injury deaths (​World Health
Organization [WHO], 201​2) and is the second leading cause of death in young people aged 15 to 29
years (​World Health Organization [WHO], 201​4). This makes suicide prevention a major public health
concern (Turecki & Brent, 2016​)​. According to a meta-analysis of 365 studies, among the most
important risk factors for suicide attempts and deaths are previous self-injurious behaviors and
suicidal ideation ​(Franklin et al., 2017)​. Suicidal ideation includes any thoughts about suicide such as
a desire for or planning of a suicide attempt and must be distinguished from actual suicidal attempts
which involve acting on these thoughts (Beck, Kovacs, & Weissman, 1979). This is addressed by item
9 of the depression module of the Patient Health Questionnaire (PHQ-9) as “thoughts that you would
be better off dead or of hurting yourself in some way” ​(Kroenke & Spitzer, 2002)​. Importantly, there is
a moderately strong association between suicide and suicidal ideation, making it an important factor
to consider when assessing suicide risk (McHugh et al., 2019; Hubers et al., 2016). It is important to
note that this association is heterogeneous and has low positive predictive value and sensitivity
(McHugh et al., 2019). As such, it is clear that not all patients who later die by suicide will express
suicidal ideation. On the other hand, suicidal ideation is much more common than attempts, and many
patients who express suicidal ideation do not actually attempt suicide (Srivastava & Kumar, 2005).
Regardless, proactive detection of ideation is helpful in the identification of patients at risk of suicide.

In current clinical practice, the primary method for identifying the presence of suicidal ideation
is through direct questioning or patient self-report. Suicidal ideation can be also be identified and
characterized using the instruments, such as the PHQ-9 or the Scale for Suicide Ideation. This
method is limited because patients may conceal suicidal intentions from clinicians, and additionally,
clinicians often fail to even ask about suicidal ideation ​(Bongiovi-Garcia et al., 2009)​. It would
therefore be clinically useful to identify which patients may be at risk of suicidal ideation without
needing to ask them directly, perhaps by using an automated screening system incorporated into the
electronic medical record, as this would allow clinicians to identify patients who might benefit from
further assessment and resources.

In the current literature, the vast majority of studies focus on identifying individual predictors
or an interaction of only a few factors, resulting in small effect sizes with low predictive value ​(Franklin
et al., 2017)​. As such, it may be useful to employ more sophisticated methods that can consider a
large number of factors when making predictions. Machine learning, which allows for the creation of
models that can consider many factors and identify complex relationships between them, may be an
ideal tool for identifying people with suicidal ideation. While a few machine learning models have been
created to predict suicide attempts (DelPozo-​Banos et al., 2018, Passos et al., 2016, Walsh et al.,
2017), we found only one that aimed at predicting suicidal ideation (Jordan et al., 2018). This study
investigated suicidal ideation in a primary care patient sample, as a significant number of people who
die by suicide have contact with primary healthcare providers in the month and year prior to their
suicide (45% and 75%, respectively) ​(Turecki & Brent, 2016; Jordan et al., 2018). Jordan and
colleagues’ model found that assessing four of the PHQ-9 items was sufficient to predict the presence
of suicidal ideation.

Our objective was to train a model to predict suicidal ideation in the general population, thus
broadening the scope of application by including potential suicide victims who would not seek medical
attention prior to their suicide attempt or who have infrequent contact with clinicians. With this goal,
we chose to use a deep learning model for a number of reasons. Firstly, deep learning models can be
robust to missing data (Cai et al., 2018), which is common in clinical datasets. More importantly, these

2
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

models are designed to find complex, non-linear patterns in data without requiring us to specify
mediators or moderators, allowing us to better approximate the intricate relationships between the
multitude of variables that put an individual at risk for suicidal thoughts.

Ideally, our prediction model would be paired with a clinical decision support system (CDSS)
that alerts clinicians and other healthcare practitioners to patients who may require further
assessment and monitoring of possible suicidal thoughts. Such a tool would connect patients with
their clinicians, allowing patients to fill out requested questionnaires and track their progress, while
providing clinicians with an organized interface to follow each of their patients and their individual
profiles. Similar tools have been found to be clinically useful in detecting and reducing sepsis
mortality, and predicting oral cancer recurrence ​(Exarchos, Goletsis, & Fotiadis, 2012; Manaktala &
Claypool, 2017)​.

Additionally, we hoped to use our machine learning approach to elucidate which patient
characteristics are involved in determining the risk for suicidal ideation. This is important not only from
the clinical perspective - that is, for understanding the factors that might cause suicidal ideation in an
individual person - but also from the public health perspective, as we may discover risk factors for
suicidal ideation amenable to intervention via social programs.

2. Methods

2.1 Dataset
The ​Canadian Community Health Survey - Mental Health Component data was collected in
2012 cross-sectionally for 25,113 people of ages 15 and over living in the ten provinces of Canada.
The data was collected either by telephone or in person and 582 data points were collected per
respondent. Participants were asked about the presence of suicidal ideation in their lifetime and in the
last twelve months. We attempted to predict participant answers to each of these questions
separately. We included only subjects who gave a firm “yes” or “no” to the questions about suicidal
ideation to maximize the discriminative ability of our model. Other responses were “not applicable”,
“don’t know”, “refusal” and “not stated”. This reduced our sample size for the prediction of lifetime
suicidal ideation to 23,859 with 21,597 responding “no” and 2,262 responding “yes” and the sample
size for the 12 months suicidal ideation prediction to 3,441 with 2,512 responding “yes” and 929
responding “no”. The size and makeup of both these subsets of the data are summarized in Table 7.
There were 485 people who responded “yes” to both questions.

2.2 Neural Network

The neural network used was a feed-forward fully-connected network with three hidden layers
of 400 neurons each activated by the scaled exponential linear unit (SELU) function (Klambauer et al.,
2017). SELU activation paired with AlphaDropout (Klambauer et al., 2017) maintains a
self-normalizing property of the trained parameters of the network so as to keep the training
procedure stable. Adam (Kingma & Ba, 2015) optimization was used to train the network. The final
prediction layer had a softmax activation, allowing the network to establish its prediction in the form of
a probability for both output classes.

2.3 Approach
In order to obtain a model that could be implemented in a real clinical environment, reducing
the number of input features to pinpoint the most important features in the dataset was necessary- as
a model that required too many input features would present challenges for data collection in the clinic
when trying to apply the model to a given patient rapidly and efficiently2. The techniques used for

2
A note on terminology: “feature” here refers to an input variable (i.e. one survey item).
3
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

feature selection involved both expertise in the field (i.e. expert feature reduction) and allowing the
model to highlight which features were the most important (Guyon & Elisseeff, 2003). A clinician
(D.B.) went through all 582 features and discarded the features which were either administrative (i.e.
redundant case identification codes or different ways of asking the same question) or which were not
reasonable to collect clinically (such as detailed health care service satisfaction metrics which would
not be appropriate in a screening context where the patient has not yet experienced services fully).
This reduced the feature set size to 196.We further reduced the number of features using machine
learning techniques. This involved analyzing the receptive fields of the trained model’s first layer and
removing “unimportant” features. Feature “importance” was calculated via the weights that the neural
network applied to a particular feature (Coates & Ng, 2011). Two cases were examined, one in which
100 features were removed, leaving 96 features in the model, and one in which 175 features were
removed, leaving 21 features in the model. We chose to remove 100 and 175 features respectively,
since the 100 feature removal didn’t affect the performance too much from the larger feature set sizes
(> 100 features) and stopped at 175 because removing any more features would cause the
performance to deteriorate vastly. The larger models were produced in order to maximize the
identification of important features and to maximize model accuracy; the smaller models were
produced in order to generate clinically tractable models with few enough questions that they could be
integrated into a standard screening assessment. Separate models were produced for both lifetime
and last 12 months suicidal ideation prediction.

In order to adjust our model to the large class imbalance that existed between the “no” and
“yes” responders, we used undersampling. The number of examples in the majority (“no”) class was
equated to the number in the minority (“yes”) class. In the case of lifetime prediction, 2,262 random
examples from the “no” class were randomly chosen for the training set to match the 2,262 samples
from the “yes” class. The class-balanced training set was then divided into 10 different random folds,
and the model was trained on 9 of these folds, leaving the final fold and all of the other 19,355 “no’s”
to serve as the validation set. This process was repeated 10 times with mutually exclusive validation
and training sets, and we noted the average of the test metrics of all runs on the validation set. It is
important to mention here that our validation set comprised of a relatively lower count of respondents
in the “yes” class compared to the initial distribution of the data, making it much harder for the model
to be able to classify respondents in the “yes” class correctly. The same sort of division was
performed for the last 12 months data using the data distribution shown in Table 7.

Given that classifying an individual to ​not have suicidal ideation when they actually are
experiencing suicidal ideation is a more costly error than predicting the inverse, we penalized false
negatives harder than other classifications. Penalization was achieved by summing to the cost
function the penalty factor defined by the number of false negatives to the power of five.

All analyses were done using the Vulcan software package (see software note). Figure 1
represents the steps taken to produce the results for this analysis.

Results

Tables 1-4 show the features used for the prediction of lifetime suicide ideation (Table 1: 96
features, Table 2: 21 features) and suicide ideation during the past 12 months (Table 3: 96 features,
Table 4: 21 features). These features are those that remain following expert feature reduction (manual
feature removal using domain expertise) and using the network’s first layer receptive fields to remove
additional features until 96 and 21 features remained for both lifetime and last 12 months suicidal
ideation models. In terms of measurement, we chose to use the AUC (area under the receiver
operating curve) as our main metric of model performance, and we also calculated the sensitivity,
specificity, negative predictive value (NPV) and positive predictive value (PPV) for each model. Tables

4
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

5 and 6 show the 10-fold cross validated results for the lifetime (96 features - 0.7983 AUC; 21
features - 0.7550 AUC) and last 12 months (96 features - 0.7611 AUC; 21 features - 0.6913 AUC)
datasets, respectively. Random forest classifiers were produced as a non-deep learning baseline;
these generally performed poorly in terms of AUC (Tables 5 and 6). In total, we produced four model
configurations : 96 and 21 features for predicting lifetime suicidal ideation and 96 and 21 features for
predicting suicidal ideation in the last 12 months.

In order to gain insight into how different features affected model predictions (i.e. feature
directionality), we performed a feature sensitivity analysis for the 21 feature models. We chose not to
perform the same analysis for the 96 feature models as it would be unsuitable to interpret due to size.
We explored how variations in values for a specific feature affected the final model prediction. We
accomplished this by iterating through all possible unique values (up to a maximum of 20 values) for
each feature and imputed all response samples to have this value. We then ran a test to determine
how many of the samples would be classified as having suicidal ideation by the model. The rightmost
columns in the 21 feature tables (Tables 2, 4) show the value of the feature where the model
predicted the most amount of suicidal ideation followed by the feature value with the lowest amount of
suicidal ideation. In Tables 2 and 4, the number in brackets next to each feature value shows the
number of examples in the test set classified as having suicidal ideation (19,788 samples in the test
set for lifetime; 1,769 for the past 12 months). This allows for some insight into the inner workings of
the neural network model. For example, in the lifetime prediction of suicidal ideation, if all the answers
to the question “have people to count on in an emergency” are set to “strongly disagree”, then 8,046
people are predicted to have suicidal ideation; this number drops to 5,158 people if the answers are
all changed to “strongly agree”.

Discussion

Here we illustrate that using our method, suicidal ideation data from the general populattion
can identify people at high risk for suicide, who could likely benefit from more in-depth screening and
resources in the context of suicide prevention.

Jordan et al. (2018) found that using only four items of the PHQ-9 provided the most accurate
predictions of suicidal ideation in their patient sample - those assessing “feelings of
depression/hopelessness, low self-esteem, worrying, and severe sleep disturbances” (Jordan et al.,
2018). Although the PHQ-9 was not included in our dataset, our model similarly found some high
impact variables related to depression, hopelessness and worrying. For instance, a high score on the
Kessler Psychological Distress Scale (K6), which assesses feelings of depression and hopelessness,
seems to be a significant risk factor for suicidal ideation (Tables 2 and 3). Unlike the Jordan model,
ours did not identify sleep problems to be a significant risk factor for suicidal ideation. One possible
explanation accommodating our results and those in the literature is that sleep problems, rather than
being a risk factor themselves, may act as a proxy for actual interacting risk factors. When such
factors are included in the data and processed by a complex model, sleep disorder factors are
rendered “irrelevant”. We will seek to verify this hypothesis in other datasets with more robust
measures of sleep. While Jordan’s model identified low self-esteem as a risk factor, our dataset
unfortunately did not contain a self-esteem variable. Our model yielded additional predictive factors
that do not overlap with those found by the Jordan team. Generalized anxiety disorder, for example,
appears to be an important predictor of suicidal ideation (Tables 1, 2, and 3). This is to be expected,
since previous research has identified anxiety disorders, including generalized anxiety disorder, as
independently predictive of suicidal ideation ​(Bentley et al., 2016; Sareen et al., 2005)​. Importantly,
our method yielded predictors related to early sexual experiences and sexual abuse. Sexual
experiences before the age of 16, including non-consensual experiences, appear to be important risk

5
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

factors for suicidal ideation (Tables 2, 3 and 4). This finding is supported by previous research linking
increased suicidal ideation and suicide attempts to early sexual abuse (Basile et al., 2006, Bedi et al.,
2011, Lopez-Castroman et al., 2013, Thompson et al., 2018, Ullman et al., 2009), thus confirming our
model’s capacity to identify known risk factors of suicidal ideation. Childhood sexual abuse is a
particularly important consideration in suicide prevention. There is extensive literature suggesting that
early-life adversity, including childhood sexual abuse, is an important predictor of suicidal behavior
(Turecki & Brent, 2016; Wanner et al., 2012; Brezo et al., 2008).

We separated prediction of suicidal ideation occuring in the last 12 months and throughout
the lifetime to disambiguate more specific short term from long term predictors. Identification of
protective factors and risk factors for both conditions may improve methods of identifying and treating
those at risk of attempting suicide. Lifetime factors may be useful in developing more long term
suicide prevention strategies, while factors predicting suicide ideation in the last 12 months can inform
the identification and treatment of patients at more immediate risk. While all predictors were related to
wellbeing, mental health, early sexual experiences and sexual abuse, we found important differences
between risk factors and protective factors for the lifetime and last 12 months conditions. Features
related to social support, such as marital status and having relationships that recognize competence
and skill, seem to be more influential in predicting suicidal ideation in the past year than throughout
the lifetime. This may indicate that measures of social support could be used to identify patients at
more immediate risk of suicidal ideation. Based on previous literature, lack of social support may be a
moderator between life stress and suicidal ideation, suggesting that a strong social support system
may be beneficial in reducing suicidal thoughts particularly during stressful times ​(Vanderhorst & Dr,
2005; Yang & Clum, 1994)​. Additionally, the level to which one has been affected by their health
problems in the previous 30 days, and dissatisfaction with life in general may be specific risk factors
for suicidal ideation in the past 12 months. Both of these measures could be included as screening
questions to identify patients who may be experiencing suicidal thoughts. By contrast, physical and
mental health related features may have more long term effects on suicidal ideation because more
health related features appear in the model predicting lifetime occurrences. As noted, depression and
anxiety symptoms were important predictors of lifetime suicidal ideation. This may also be related to
the timing of the data collection, as a smaller number of respondents would have been experienced a
mood episode or high levels of anxiety during the interview year itself than over the course of their
lifetimes.

We identified several surprising features that did not show a clear directionality in our
sensitivity analysis. While these features are not clear risk or protective factors, they seem to interact
with other features to predict suicidal ideation. Notably, features related to smoking, including when
patients started and stopped smoking, appeared in both the lifetime and past 12 months prediction
models as potential moderators of suicidal ideation. Previous explorations found that cigarette use
increases the risk of suicidal ideation, a relationship that could potentially be explained by the lower
levels of serotonin found in smokers ​(Malone et al., 2003; Tanskanen, Viinamäki, Hintikka,
Koivumaa-Honkanen, & Lehtonen, 1998)​. Our results complement these findings, while alluding to a
more complex relationship without clear directionality when other factors are considered. This
supports a focus on public health and public mental health interventions on smoking. Additionally,
contact with the police may moderate suicidal ideation, highlighting a need to follow up with people
who may have had a traumatic experience leading to police intervention, or negative interactions with
the police (DeVylder et al., 2018).

We identified several predictors that are easy to obtain, including sociodemographic features.
Interestingly, Jordan et al. did not find sociodemographic features useful in the prediction of suicidal
ideation (2018), but as we were using a census dataset with a large and varied array of
sociodemographic features, we were able to identify more predictors amongst them that would be

6
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

amenable to upstream intervention. As opposed to more expensive data like neuroimaging and
genetic testing, sociodemographic predictors can be very useful in clinical practice, especially with
respect to screening, since they are easily accessible to healthcare professionals through direct
questioning or self-report questionnaires.

As can be seen in Tables 5 and 6, the 96 feature models have higher AUCs. This is expected,
as the network is able to make better predictions when it has more information of the different patients
it is classifying. It is worthwhile to discuss the pros and cons of having larger or smaller models. Large
models which do not overfit allow us to identify more predictors which may be modifiable and are
therefore potentially useful from a public health standpoint. Smaller models are easier to implement
because patients need to answer fewer questions in order to provide the model with sufficient
information to make a prediction. Thus, there exists an interesting trade-off between model accuracy
and ease of data acquisition upon selecting the number of features to include.. For example, the
difference in the AUC for the last 12 months model presented here is 0.69 for the 21 feature model vs.
0.76 for the 96 feature model. Does this 7 point difference justify a larger model that is more accurate
but more difficult to collect? While 7 points may only seem like a moderate difference, when
considering predictions on a population scale we might expect a significant difference in the absolute
number of people correctly classified. Implementation of models such as these will hinge on finding
the right balance between model complexity and accuracy in order to provide models that are both
meaningful and feasible to implement.

It is also important to note the high negative predictive values (NPV) of our predictions. This
metric indicates that the network is almost always correct when it classifies an example as not having
suicidal ideation. This is crucial, as it signifies the utility of our model in helping, alongside good
clinical judgement and history taking, to rule out suicidal ideation in populations matching those in the
dataset. Given that currently, clinicians have difficulty ruling out suicidal thinking or risk (McDowell et
al., 2011), such a tool would be clinically useful. This must be balanced against the risk of false
positives, which can lead to unnecessary intervention and confinement, as well as against the fact
that the absence of suicidal ideation at a single point in time does not rule out the risk of suicide
(McHugh et al., 2019). However, given that this model predicts suicidal ideation and not risk of
attempt, a positive result could be used to open a conversation between a clinician and patient, which
might lead to more appropriate assessment and treatment before the risk of an attempt increases.
This in turn may become a useful approach for the prevention of suicide via upstream identification of
at-risk patients in the general population, though this remains speculative and should be expanded on
in future work exploring factors that predict conversion of ideation to action.

There are several limitations to our current work. While using an interview-based census
dataset allows for a large sample size in the general population, it does mean that there is no
independent verification of participant responses or any clinician-rated scales. Our use of deep
learning provides for a powerful technique that outperforms random forest classifiers, but which is
generally less easy to interpret than other machine learning techniques; that being said, our sensitivity
analysis does allow some insight into the model parameters which could be further evaluated using
classical statistics.

It is worth discussing the practical implementation of a predictive tool of suicidal ideation in

clinical practice, as this would bring both possible benefits and challenges. One possible
implementation of this tool would be as an automated screening tool integrated into electronic medical
records in emergency departments or outpatient clinics. Benefits - which would need to be verified in
clinical studies - could include earlier and more accurate identification of suicidal ideation, which
would lead to more patients being offered appropriate services, such as access to a therapist or to
crisis resources. This in turn would hopefully lead to a reduction in the number of patients making

7
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

suicide attempts or completing suicide, though this would depend on the efficacy of the offered
interventions. Nonetheless, challenges and potential dangers exist. Models that predict suicidal
ideation could be used by some clinicians to justify interventions such as forced hospitalization, which
raises serious concerns about the effect of implementing such models on patient autonomy and
clinician medico-legal risk. In addition, it is unclear what effect having an automated screening tool for
suicidal ideation would have on clinician behavior. It might improve clinician awareness of the
importance of screening for and offering support to patients with suicidal ideation; at the same time, it
may reinforce the habit of many clinicians to avoid asking about suicidal ideation, fostering an
over-reliance on an imperfect system to screen for a potentially serious clinical phenomenon. Any
implementation of such a screening system would require significant investment in the training of
clinicians and should be accomplished in partnership with patient and clinician representatives.

Appendix

Table 1. Features retained in the 96 feature version of the lifetime suicidal ideation prediction model

Sociodemographic
White or non-white race/visible minority
Occupation group
Currently pregnant
Currently attending school, college, CEGEP, or university
Time in Canada since immigration
Employment status last week
World Health Organization (WHO) Disability Assessment Schedule (WHODAS) score
Lifestyle
Number of cigarettes smoked per day (former daily smokers)
Type of smoker
Body Mass Index (BMI) (self-reported)
Experienced drug abuse or dependence (including cannabis) in their lifetime
Self-rated physical health
Experienced alcohol abuse or dependence in their lifetime
Experienced alcohol abuse in their lifetime
Type of smoker (calculated)3
Self-perceived rating of ability to handle day-to-day demands
Importance of religious or spiritual values in daily life
Number of years since stopped smoking daily (former daily smokers)
Self-perceived personal ability to deal with stress
Has ever smoked a whole cigarette

3
This question was computed from the results of other questions about smoking: if the respondent
has smoked 100 or more cigarettes during their lifetime, if the respondent has ever smoked a whole
cigarette, what type of smoker the respondent is, if the respondent has ever smoked cigarettes daily.
This calculation produces a 1 (daily smoker) to 6 (never smoked) score.
8
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Extent to which religious or spiritual values provide strength to face everyday difficulties
Average number of hours of moderate or vigorous physical activity in the past week
Satisfaction with life in general
Past Experiences or Trauma
Had contact with the police in the past 12 months as a victim of a crime
Had contact with the police in the past 12 months as a witness to a crime
Had contact with the police in the past 12 months for work
Had contact with the police in the past 12 months for other4 reason
Before age 16, sought assistance from a child protection organization for difficulties at home
Psychiatric Diagnoses or Symptoms
Received professional treatment during the past year for symptoms of generalized anxiety
disorder
Has a learning disability
Has attention deficit disorder
Personal impact scale of mental health experiences5
Duration of the longest episode of major depressive disorder
Consulted a psychologist for emotional/mental health/alcohol/drug problems in the past 12
months
Consulted a psychiatrist for emotional/mental health/alcohol/drug problems in the past 12
months
Experienced an episode of major depressive disorder in the past 12 months
Duration of longest episode of generalized anxiety disorder
Duration of longest episode of generalized anxiety disorder
Experienced an episode of major depressive disorder in their lifetime
Interference of generalized anxiety disorder on daily activities and responsibilities in the
past 12 months6
Screened in depression7
Interference of mania on daily activities and responsibilities in the past 12 months8
Has a mood disorder9

4
Reason other than being the victim of a crime, a witness to a crime, for work, for a public information
session, due to a traffic violation or accident, or due to a family member’s emotional, mental health,
alcohol or drug problems
5
On a scale of 0 (lowest recordable personal impact of stigma experiences) to 60 (highest recordable
personal impact of stigma experiences)
6
On a scale of 0 (low interference) to 10 (high interference)
7
This question was computed from the results of other screening questions for depression: if the
respondent ever had a period of days when they felt sad, empty or depressed for most of the day; if
the respondent ever had a period of days when they felt very discouraged over how life was going; if
the respondent ever had a period of days when they lost interest in things they usually enjoy.
8
On a scale of 0 (low interference) to 10 (high interference)
9
Depression, bipolar disorder, mania or dysthymia
9
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Interference of the major depressive episode on daily activities and responsibilities in the
past 12 months10
Has been treated for an emotional or mental health problem in the past 12 months
Screened in mania11
Has tested positive for generalized anxiety disorder in the past 12 months
Has an anxiety disorder such as phobia, panic disorder or obsessive-compulsive disorder
Experienced bipolar II in their lifetime
Screened in general anxiety disorder12
Experienced generalized anxiety disorder in their lifetime
Has used antipsychotic medications in the past 2 days
Duration of the longest episode of mania
Amount of difficulty in day to day work or school activities in the past 30 days
Amount of difficulty concentrating on tasks in the past 30 days
Total number of medications used in the past 2 days
Medical Comorbidities
Has chronic fatigue syndrome
Has high blood pressure
Has heart disease
Has a chronic condition
Has diabetes
Has ever had cancer
Suffers from multiple chemical sensitivities
Ever diagnosed with high blood pressure
Has or was previously diagnosed with high blood pressure
Extent to which the respondent has been emotionally affected by their own health problems
in the past 30 days
Amount of difficulty standing for long periods in the past 30 days
Social Support
Amount of time spent in an average week providing help to family member(s)
Family member(s)’s problems cause respondent worry, anxiety or depression

10
On a scale of 0 (low interference) to 10 (high interference)
11
This question was computed from the results of other screening questions for mania: if the
respondent ever had a manialike period lasting several days or longer; if the respondent ever had a
period of days when they were so irritable that they either started arguments, shouted at people or hit
people.
12
This question was computed from the results of other screening questions for generalized anxiety
disorder: if the respondent ever had a time when they worried much more about things than other
people with the same problems; if the respondent ever had a time when they were much more
nervous or anxious than most other people with the same problems; if the respondent ever had a
period of 6 months or more when they were anxious or worried most days.
10
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Family member(s)’s problems cause respondent embarrassment

How often the respondent felt that others did things that were thoughtless or inconsiderate
in the past month
How often the respondent felt that others were critical of their behaviour in the past month
How often the respondent felt that others made too many demands of them in the past
month
Regular contact with people detrimental to respondent's wellbeing due to discomfort and
stress
Impact of negative opinions or unfair treatment on housing situation13
Has close relationships
Help provided by respondent to family member(s) by providing company or emotional
support
Has a friend who has ever been treated for an emotional or mental health problem
Impact of negative opinions or unfair treatment on financial situation
Has people to count on in emergency
Marital status
Help provided by respondent to family member(s) by spending time doing other things14
related to their problems
Help received in the past 12 months
Any family member has emotional, mental health, alcohol or drug problems
Number of family members with emotional, mental health, alcohol or drug problems
Has people who admire their talents and abilities
Has people who enjoy the same activities
Has someone to talk to about important decisions
Belongs to a group or people who share attitudes and beliefs
Has a trustworthy person to turn to for advice
Family member has been treated for emotional or mental health in the past 12 months
Amount of difficulty joining in community activities in the past 30 days
Other
Respondent alone during interview
Interview by telephone or in person

13
On a scale of 0 (has not been affected) to 10 (has been severely affected)
14
All things other than providing practical support, company or emotional support
11
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Table 2. Features retained in the 21 feature version of the lifetime suicidal ideation prediction model.

Results of sensitivity analysis expressed as total numbers and ratios are presented in the middle and
right columns. These express how many people would be classified as having suicidal ideation if all
respondents tested gave answers at one or another extreme within the value range for a given
question. For example, in row one, if all tested participants answered that they had stopped smoking
less than 1 year ago, then there would be 6,819 positive classifications of suicidal ideation, and this
would drop to 5,195 if all samples.had stopped smoking 3 or more years ago. The right most column
describes the ratio of these two numbers.

Feature Description Model Sensitivity: Possible Model Sensitivity: Ratios

Response (number of
samples classified as having
suicidal ideation)
Sociodemographic
Lifestyle
Opinion of own weight (self-reported) Overweight (6,362), Just about Overweight vs. Just about right
right (4,799) (1.33 : 1)
When stopped smoking completely Less than 1 year (6,819), 3 or Less than 1 year vs. 3 or more
(former daily smokers more years (5,195) years: (1.31 : 1)
Has experienced alcohol abuse in Yes (6,452), No (5,220) Yes vs. No (1.24 : 1)
their lifetime
Ever smoked cigarettes daily Yes (6,608), No (6,160) Yes vs. No (1.07 : 1)
Average number of hours of 13.5 (5,998), 0 (5,144) 13.5 hours vs. 0 hours of
moderate or vigorous physical activity physical activity (1.06 : 1)
in the past week
Used any medications for mental No (5,398), Yes (5,145) No vs. Yes: (1.05 : 1)
health, alcohol or drugs in the past 2
days
Engaged in moderate or vigorous Yes (5,607), No (5,334) Yes vs. No (1.05 : 1)
physical activity in the past week
Past Experiences or Trauma
Number of times before age 16 More than 10 times (9,677), More than 10 times vs. Never:
experienced sexual touching Never (4,790) (2.02 : 1)
Psychiatric Diagnoses or
Symptoms
Score on the Kessler Psychological Highest Recordable Distress Highest vs. Lowest Recordable
Distress Scale (K6) in the past month (19,758), Lowest Recordable Distress (12.79 : 1)
Distress (1,545)
Screened in generalized anxiety Yes (7,663), No (3,219) Yes vs. No (2.38 : 1)
disorder
Personal impact scale of mental Lowest recordable personal Lowest recordable personal
health experiences15 impact of stigma experiences impact of stigma experiences
(15,520), Highest recordable vs. Highest recordable
personal impact of stigma personal impact of stigma
experiences (8,229) experiences (1.89 : 1)

15
On a scale of 0 (lowest recordable personal impact of stigma experiences) to 60 (highest recordable
personal impact of stigma experiences)
12
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Has tested positive for generalized No (5,391), Yes (4,016) No vs. Yes (1.34 : 1)
anxiety disorder in the past 12
months
Has post-traumatic stress disorder Yes (6,983), No (5,384) Yes vs. No (1.17 : 1)
Duration of the longest episode of 5 years or more (2,385), 2 5 years or more vs. 2 weeks
major depressive disorder weeks (2,052) (1.16 : 1)
Interference of depression on daily 0 (7,290), 9.6 (7,102) Score of 0 vs score of 9.6
activities and responsibilities in the (1.02 : 1)
past 12 months
Medical Comorbidities
Has or was previously diagnosed with No (5,445), Yes (4,654) No vs. Yes (1.16 : 1)
cancer
Has ever been diagnosed with high Yes (6,315), No (5,793) Yes vs. No (1.09 : 1)
blood pressure
Has or was previously diagnosed with Yes (5,424), No (5,397) Yes vs. No (1.01 : 1)
high blood pressure
Social Support
Has a trustworthy person to turn to Strongly disagree (9,350), Strongly disagree vs. strongly
for advice Strongly agree (4,898) agree (1.91 : 1)
Has people to count on in emergency Strongly disagree (8,046), Strongly disagree vs. strongly
Strongly agree (5,158) agree (1.56 : 1)
Impact of negative opinions or unfair Has not been affected Has not been affected vs. has
treatment on housing situation (18,631), Has been severely been severely affected (1.06 :
affected (17,508) 1)

Table 3. Features retained in the 96 feature version of the last 12 months suicidal ideation prediction
model.

Sociodemographic
Age
Sex
Currently pregnant
Full-time or part-time working status
Occupation group
Highest level of education attained by any member of household
Highest level of education attained by respondent
Has insurance that covers all or part of the cost of prescriptions
Lifestyle
Satisfaction with life in general
Self-rated mental health
Average number of hours of moderate or vigorous physical activity in the past week
Self-perceived personal ability to deal with stress
Has smoked 100 or more cigarettes during their lifetime

13
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Age when first smoked a whole cigarette

Age when started smoking daily
Ever smoked cigarettes daily
Age when started smoking daily (former daily smokers)
Number of cigarettes smoked per day (former daily smokers)
Number of years since stopped smoking daily (former daily smokers)
Type of smoker
Number of years smoked
Experienced alcohol abuse or dependence in their lifetime
Experienced drug abuse or dependence (excluding cannabis) in their lifetime
Extent to which difficulties mentioned in the WHO Disability Assessment Schedule
(WHODAS) 2.016 interfere with life
Extent to which religious or spiritual values provide strength to face everyday difficulties
Most important source of feelings of stress
Past Experiences or Trauma
Had contact with the police in the past 12 months at a public information session
Had contact with the police in the past 12 months as a victim of a crime
Had contact with the police in the past 12 months for work
Number of times before age 16 saw or heard caregiver hit another adult
Number of times before age 16 was pushed, grabbed, shoved or thrown at by adult
Number of times before age 16 experienced forced or attempted forced sexual activity
Number of types of childhood maltreatment experienced
Psychiatric Diagnoses or Symptoms
Screened in depression17
Has a mood disorder18
Has mood disorder other than depression, bipolar disorder, mania or dysthymia
Has a learning disability
Has attention deficit disorder
Score on the Kessler Psychological Distress Scale (K6) in the past month
Ever consulted a medical doctor or other professional about symptoms of depression

16
Difficulty standing for long periods, difficulty taking care of household responsibilities, difficulty
learning a new task, difficulty joining in community activities, emotionally affected by health problems,
difficulty concentrating, difficulty walking a long distance, difficulty washing their whole body, difficulty
getting dressed, difficulty dealing with people they do not know, difficulty maintaining a friendship,
difficulty in day to day work or school activities
17
This question was computed from the results of other screening questions for depression: if the
respondent ever had a period of days when they felt sad, empty or depressed for most of the day; if
the respondent ever had a period of days when they felt very discouraged over how life was going; if
the respondent ever had a period of days when they lost interest in things they usually enjoy.
18
Depression, bipolar disorder, mania or dysthymia
14
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Received professional treatment in the past year for symptoms of depression

Experienced an episode of major depressive disorder in their lifetime
Duration of the longest episode of major depressive disorder
Whether or not the major depressive episode interfered significantly in daily activities and
responsibilities in the past 12 months
Experienced mania in their lifetime
Age at occurence of most recent episode of manic feelings
Duration of the longest episode of mania
Whether or not mania interfered significantly in daily activities and responsibilities in the
past 12 months
Consulted a doctor/other professional during the past year for symptoms of generalized
anxiety disorder
Duration of the longest episode of generalized anxiety disorder
Has tested positive for generalized anxiety disorder in the past 12 months
Interference of generalized anxiety disorder on daily activities and responsibilities in the
past 12 months19
Whether or not generalized anxiety disorder interfered significantly in daily activities and
responsibilities in the past 12 months
Consulted a psychologist for emotional/mental health/alcohol/drug problems in the past
12 months
Used benzodiazepines in the past 2 days
Used antipsychotics in the past 2 days
Used medications other than antidepressants, benzodiazepines and antipsychotics in the
past 2 days
Amount of difficulty concentrating on tasks in the past 30 days
Medical Comorbidities
Has asthma
Has arthritis
Has back problems excluding fibromyalgia and arthritis
Has high blood pressure
Has chronic bronchitis, emphysema or chronic obstructive pulmonary disease (COPD)
Has heart disease
Suffers from the effects of a stroke
Has a bowel disorder, Crohn's Disease or ulcerative colitis
Suffers from multiple chemical sensitivities
Has a chronic condition
Has or was previously diagnosed with high blood pressure

19
On a scale of 0 (low interference) to 10 (high interference)
15
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Degree of pain usually felt by the respondent and whether it prevents them from
performing certain activities
Amount of difficulty standing for long periods in the past 30 days
Amount of difficulty walking long distances in the past 30 days
Social Support
Can count on people they know to help deal with their most important source of stress
Has a family member who has ever been treated for emotional or mental health
Impact of negative opinions or unfair treatment on family relationships20
Impact of negative opinions or unfair treatment on romantic life21
Impact of negative opinions or unfair treatment on work or school life22
Impact of negative opinions or unfair treatment on housing situation23
Any family member has emotional, mental health, alcohol or drug problems
Extent to which the respondent's life is affected by family member(s)’s emotional, mental
health, alcohol or drug problems
Help provided by the respondent to their family member(s) with practical things
Help provided by the respondent to their family member(s) by spending time doing other
things related to their problems
Amount of time spent in average week providing help to family member(s)
Has people to depend on
Has people who enjoy the same activities
Has close relationships
Has someone to talk to about important decisions
Has people who admire their talents and abilities
Regular contact with people is detrimental to their wellbeing due to discomfort and stress
How often the respondent felt that others made too many demands of them in the past
month
How often the respondent felt that others were critical of their behaviour in the past
month
How often the respondent felt that others did things that were thoughtless or
inconsiderate in the past month
How often the respondent felt that others acted angry or upset with them in the past
month
Amount of negative social interactions
Other
Respondent was alone during the interview

20
On a scale of 0 (has not been affected) to 10 (has been severely affected)
21
On a scale of 0 (has not been affected) to 10 (has been severely affected)
22
On a scale of 0 (has not been affected) to 10 (has been severely affected)
23
On a scale of 0 (has not been affected) to 10 (has been severely affected)

16
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Answers were affected by the presence of another person

Table 4. Features retained in the 21 feature version of the last 12 months suicidal ideation prediction
model.

Results of sensitivity analysis expressed as total numbers and ratios are presented in the middle and
right columns.

Feature Description Model Sensitivity: Response Model Sensitivity: Ratios

(number of samples classified
as having suicidal ideation)
Sociodemographic
Sex Male (704), Female (486) Male vs. Female (1.45 : 1)
Currently attending school, Yes (654), No (565) Yes vs. No (1.16 : 1)
college, CEGEP, or
university
Lifestyle
Satisfaction with life in Very dissatisfied (1747), Very Very dissatisfied vs. Very satisfied
general satisfied (233) (7.50 : 1)
Extent to which the Extreme (1421), None (442) Extreme vs. None (3.21 : 1)
respondent has been
emotionally affected by their
own health problems in the
past 30 days
How long ago stopped 3 or more years (467), less than 1 3 or more years vs. Less than 1
smoking (former daily year (341) year (1.37 : 1)
smokers)
Number of years since 11 or more years (314), 3 to 5 11 or more years vs. 3-5 years
stopped smoking (never years (230) (1.36 : 1)
daily smokers)
How long ago stopped Less than 1 year (867), 3 or more Less than 1 year vs. 3 or more
smoking (never daily years (643) years (1.35 : 1)
smokers)
Experienced alcohol abuse Yes (597), No (569) Yes. vs. No (1.05 : 1)
or dependence in their
lifetime
Age when started smoking 45-49 years and 50 years or more 45-49 years and 50 years or more
daily (daily smoker) (both 577), 12-14, 15-17, vs. 12-14, 15-17, 18-19, 20-24,
18-19, 20-24, 25-29 (all 575) 25-29 (1.00 : 1)
Number of years smoked 3,6,9,48,51,54,57,60,63,66,69,72, 3,6,9,48,51,54,57,60,63,66,69,72,
77 (all 573), 0, and all 77 vs. 0, and all multiples of
multiples of 3 between 12 and 45 3 between 12 and 45 inclusive
inclusive (574) (1.00 : 1)
Past Experiences or
Trauma

17
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Number of times before age More than 10 times (964), Never More than 10 times vs. Never
16 experienced forced or (472) (2.04 : 1)
attempted forced sexual
activity
Had contact with the police No (583), Yes (401) No vs. Yes (1.45 : 1)
in the past 12 months at a
public information session
Had contact with the police Yes (642), No (573) No vs. Yes (1.12 : 1)
in the past 12 months as a
witness to a crime
Psychiatric Diagnoses or
Symptoms
Has attention deficit disorder Yes (793), No (557) Yes vs. No (1.42 : 1)
Used antidepressants in the Yes (599), No (570) Yes vs. No (1.05 : 1)
past 2 days
Medical Comorbidities
Social Support
Has relationships that Strongly disagree (864), Strongly Strongly disagree vs. Strongly
recognize competence and agree (511) agree (1.69 : 1)
skill
Marital status Single (632), Married (468) Single vs. Married (1.35 : 1)
Has people who admire Strongly disagree (656), Strongly Strongly disagree vs. Strongly
respondent’s talents and agree (551) agree (1.28 : 1)
abilities
Any family member has Yes (594), No family members Yes vs. No family members (1.13 :
emotional, mental health, (525) 1)
alcohol or drug problems
Help provided by respondent No (481), Yes (450) No vs. Yes (1.07 : 1)
to family member(s) with
practical things
Impact of negative opinions Has not been affected (1,589), Has not been affected vs. Has
or unfair treatment on Has been severely affected been severely affected (1.06 : 1)
romantic life (1,497)

Table 5. Lifetime and last 12 months suicidal ideation prediction model metrics, including comparison
between random forest baseline model and deep learning (cross entropy loss function) results.

Number of Loss
Sensitivity Specificity AUC PPV NPV
Features Function

Cross 0.0319
Lifetime 96 0.7059 0.7528 0.7983 0.9956
entropy 0

Random
96 0.0148 0.997 0.50 0.0148 0.997
Forest

18
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

classifier
(non-deep
learning
baseline)

Cross 0.0266
21 0.6261 0.7356 0.7550 0.9943
entropy 0

Random
Forest
classifier
21 0.0143 0.997 0.50 0.0143 0.997
(non-deep
learning
baseline)

Number of Loss Sensitivity Specificity AUC PPV NPV

Features Function

96 Cross 0.6629 0.7395 0.7611 0.1232 0.9754

entropy

96 Random 0.0784 0.983 0.497 0.0784 0.983

Forest
classifier
(non-deep
Last 12 learning
Months baseline)

21 Cross 0.6205 0.6889 0.6913 0.0994 0.9705

entropy 0

21 Random 0.0581 0.981 0.501 0.0581 0.981

Forest
classifier
(non-deep
learning
baseline)

Supplementary Materials

Table 7. Sizes of datasets and distribution of data.

Lifetime Suicidal Ideation Last 12 Months Suicidal
Ideation

Total number of examples 23,859 3,441

Number of examples that 2,262 929

answered ‘yes’

Number of examples that 21,597 2,512

answered ‘no’

Number of males 10,871 1,437

19
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Number of females 12,988 2,004

60 TO 64 YEARS 2,104 280

55 TO 59 YEARS 2,097 376

15 TO 19 YEARS 1,932 291

20 TO 24 YEARS 1,886 330

65 TO 69 YEARS 1,844 209

50 TO 54 YEARS 1,821 329

30 TO 34 YEARS 1,772 279

35 TO 39 YEARS 1,616 279

40 TO 44 YEARS 1,585 282

45 TO 49 YEARS 1,571 254

80 YEARS OR MORE 1,560 72

25 TO 29 YEARS 1,521 279

70 TO 74 YEARS 1,388 114

75 TO 79 YEARS 1,162 67

Table 8. Sensitive cost function experiment results

The rows in Table 8 have the sensitive cost function that indicate an experimental approach
where instead of using the simple cross entropy loss function, we added an extra penalty to the false
negatives, in the hope of improving the number of false negatives. As can be seen, the sensitivity did
improve slightly for the 96 and 21 feature lifetime datasets. However, it actually decreased the
sensitivity in both data subsets for the last 12 months prediction. This could have been because the
number of samples in the last 12 months dataset is extremely small, and thus the sensitive cost
function did not have the desired effect of being exposed to suboptimal levels of data variation

Number of
Sensitivity Specificity AUC PPV NPV
Features

Lifetime - 96 0.7140 0.7436 0.7995 0.03120 0.9955

Lifetime - 21 0.6369 0.7189 0.7486 0.02560 0.9941

Last 12
0.6603 0.7155 0.7494 0.1138 0.9744
months - 96

Last 12
0.5909 0.6879 0.6884 0.0946 0.9683
months - 21

20
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Methods Addendum

Variance Thresholding
Variance thresholding, a method which removes columns (i.e. features) if they do not vary
sufficiently across the patient samples depending on the threshold given, was attempted but
discontinued as it seemed to adversely affect the predictive power of the results. We presume this
may have been due to the extreme imbalance in the dataset, where columns removed via this method
may have in fact been the determining features that helped distinguish the difference between having
suicidal ideation and not having them.

Figure 1. Flow of data through our training and inference system broken into three phases, 1) Data
Processing – reduce dataset features using expert reduction & receptive fields, 2) Model Training &
Testing – 10-fold cross validation using under sampling of the ‘yes’ class and training a neural
network, and 3) Sensitivity Analysis – discovering feature directionality for our 21 feature trained
models.

Disclosure
Myriam Tanguay-Sela and Sonia Israel are employees and shareholders of Aifred Health, a
medical technology company that uses deep learning to increase treatment efficacy in psychiatry.
David Benrimoh, Robert Fratila and Kelly Perlman are shareholders of Aifred Health. All other authors
declare no conflict of interest.

21
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

References

Basile, K. C., Black, M. C., Simon, T. R., Arias, I., Brener, N. D., & Saltzman, L. E. (2006). The
association between self-reported lifetime history of forced sexual intercourse and recent
health-risk behaviors: Findings from the 2003 national youth risk behavior survey. ​Journal of
Adolescent Health​, 39(5), 752.e1-752.e7. doi:10.1016/j.jadohealth.2006.06.001
Bedi, S., Nelson, E. C., Lynskey, M. T., Cutcheon, V. V., Heath, A. C., Madden, P. A., & Martin, N. G.
(2011). Risk for suicidal thoughts and behavior after childhood sexual abuse in women and
men. ​Suicide and Life-Threatening Behavior,​ 41(4), 406-415.
doi:10.1111/j.1943-278x.2011.00040.x
Bentley, K. H., Franklin, J. C., Ribeiro, J. D., Kleiman, E. M., Fox, K. R., & Nock, M. K. (2016). Anxiety
and its disorders as risk factors for suicidal thoughts and behaviors: A meta-analytic review.
Clinical Psychology Review, 43, 30–46. https://doi.org/10.1016/j.cpr.2015.11.008
Bongiovi-Garcia, M. E., Merville, J., Almeida, M. G., Burke, A., Ellis, S., Stanley, B. H., … Oquendo,
M. A. (2009). Comparison of clinical and research assessments of diagnosis, suicide attempt
history and suicidal ideation in Major Depression. Journal of Affective Disorders, 115(0),
183–188. https://doi.org/10.1016/j.jad.2008.07.026
Brezo, J., Paris, J., Vitaro, F., Hébert, M., Tremblay, R. E., & Turecki, G. (2008). Predicting suicide
attempts in young adults with histories of childhood abuse. ​British Journal of Psychiatry,​
193(2), 134-139. doi:10.1192/bjp.bp.107.037994
Cai, L., Wang, Z., Gao, H., Shen, D., & Ji, S. (2018). Deep Adversarial Learning for Multi-Modality
Missing Data Completion. ​Proceedings of the 24th ACM SIGKDD International Conference on
Knowledge Discovery & Data Mining - KDD '18​, 1158-1166. doi:10.1145/3219819.3219963
Coates, A., & Ng, A. Y. (2011). Selecting Receptive Fields in Deep Networks. ​NIPS 2011.​
https://doi.org/10.1016/j.psychres.2009.03.008
Delpozo-Banos, M., John, A., Petkov, N., Berridge, D. M., Southern, K., Lloyd, K., . . .
Travieso, C. M. (2018). Using neural networks with routine health records to
identify suicide risk: Feasibility study (Preprint). JMIR Mental Health, 5(2).
doi:10.2196/preprints.10144
DeVylder, J. E., Jun, H., Fedina, L., Coleman, D., Anglin, D., Cogburn, C., . . . Barth, R. P. (2018).
Association of exposure to police violence with prevalence of mental health symptoms among
urban residents in the United States. ​JAMA Network Open​, 1(7).
doi:10.1001/jamanetworkopen.2018.4945
Exarchos, K. P., Goletsis, Y., & Fotiadis, D. I. (2012). Multiparametric Decision Support System for the
Prediction of Oral Cancer Reoccurrence. IEEE Transactions on Information Technology in
Biomedicine, 16(6), 1127–1134. https://doi.org/10.1109/TITB.2011.2165076
Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., … Nock, M. K.
(2017). Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of
research. Psychological Bulletin, 143(2), 187–232. https://doi.org/10.1037/bul0000084
Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection (L. P. Kaelbling,
Ed.). ​Journal of Machine Learning Research​, 3, 1157-1182.
Hubers, A. A., Moaddine, S., Peersmann, S. H., Stijnen, T., Duijn, E. V., Mast, R. C., . . . Giltay, E. J.
(2016). Suicidal ideation and subsequent completed suicide in both psychiatric and
non-psychiatric populations: A meta-analysis. ​Epidemiology and Psychiatric Sciences,​ 27(2),
186-198. doi:10.1017/s2045796016001049
Jordan, P., Shedden-Mora, M. C., & Löwe, B. (2018). Predicting suicidal ideation in primary
care: An approach to identify easily assessable key variables. ​General Hospital
Psychiatry,​ 51, 106-111. doi:10.1016/j.genhosppsych.2018.02.002
Kingma, D. P., & Ba, J. L. (2015). Adam: A Method for Stochastic Optimization. In ​International
Conference for Learning Representations​. Retrieved from https://arxiv.org/abs/1412.6980

22
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Klambauer, G., Unterthiner, T., & Mayr, A. (n.d.). Self-normalizing neural networks (I.
Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, et al., Eds.).
Advances in Neural Information Processing Systems 30 (NIPS 2017).​
Kroenke, K., & Spitzer, R. L. (2002). The PHQ-9: A new depression diagnostic and severity measure.
Psychiatric Annals; Thorofare, 32(9), 509–515.
http://dx.doi.org/10.3928/0048-5713-20020901-06
Lopez-Castroman, J., Melhem, N., Birmaher, B., Greenhill, L., Kolko, D., Stanley, B., . . . Oquendo, M.
A. (2013). Early childhood sexual abuse increases suicidal intent. ​World Psychiatry​, 12(2),
149-154. doi:10.1002/wps.20039
Malone, K. M., Waternaux, C., Haas, G. L., Cooper, T. B., Li, S., & Mann, J. J. (2003). Cigarette
Smoking, Suicidal Behavior, and Serotonin Function in Major Psychiatric Disorders. American
Journal of Psychiatry, 160(4), 773–779. https://doi.org/10.1176/appi.ajp.160.4.773
Manaktala, S., & Claypool, S. R. (2017). Evaluating the impact of a computerized surveillance
algorithm and decision support system on sepsis mortality. Journal of the American Medical
Informatics Association, 24(1), 88–95. https://doi.org/10.1093/jamia/ocw056
McDowell, A. K., Lineberry, T. W., & Bostwick, J. M. (2011). Practical suicide risk management for the
busy primary care physician. ​Mayo Clinic Proceedings​, 86(8), 792-800.
https://doi.org/10.4065/mcp.2011.0076
McHugh, C. M., Corderoy, A., Ryan, C. J., Hickie, I. B., & Large, M. M. (2019). Association between
suicidal ideation and suicide: Meta-analyses of odds ratios, sensitivity, specificity and positive
predictive value. ​BJPsych Open,​ 5(2). doi:10.1192/bjo.2018.88
Passos, I. C., Mwangi, B., Cao, B., Hamilton, J. E., Wu, M., Zhang, X. Y., . . . Soares, J. C.
(2016). Identifying a clinical signature of suicidality among patients with mood disorders: A
pilot study using a machine learning approach. J​ournal of Affective Disorders,​ ​193,​ 109-116.
doi:10.1016/j.jad.2015.12.066
Sareen, J., Cox, B. J., Afifi, T. O., Graaf, R. de, Asmundson, G. J. G., Have, M. ten, & Stein, M. B.
(2005). Anxiety Disorders and Risk for Suicidal Ideation and Suicide Attempts: A
Population-Based Longitudinal Study of Adults. Archives of General Psychiatry, 62(11),
1249–1257. https://doi.org/10.1001/archpsyc.62.11.1249
Suicide. (2018). Retrieved December 1, 2018, from World Health Organization website:
http://www.who.int/news-room/fact-sheets/detail/suicide
Tanskanen, A., Viinamäki, H., Hintikka, J., Koivumaa-Honkanen, H.-T., & Lehtonen, J. (1998).
Smoking and Suicidality Among Psychiatric Patients. American Journal of Psychiatry, 155(1),
129–130. https://doi.org/10.1176/ajp.155.1.129
Thompson, M. P., Kingree, J. B., & Lamis, D. (2018). Associations of adverse childhood experiences
and suicidal behaviors in adulthood in a U.S. nationally representative sample. ​Child: Care,
Health and Development,​ 45(1), 121-128. doi:10.1111/cch.12617
Turecki, G., & Brent, D. A. (2016). Suicide and suicidal behaviour. ​The Lancet,​ 387(10024),
1227-1239. doi:10.1016/S0140-6736(15)00234-2
Ullman, S. E., & Najdowski, C. J. (2009). Correlates of serious suicidal ideation and attempts in
female adult sexual assault survivors. ​Suicide and Life-Threatening Behavior,​ 39(1), 47-57.
doi:10.1521/suli.2009.39.1.47
Vanderhorst, R. K., & Dr, S. M. (2005). Social relationships as predictors of depression and suicidal
ideation in older adults. Aging & Mental Health, 9(6), 517–525.
https://doi.org/10.1080/13607860500193062
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017). Predicting risk of suicide attempts
over time through machine learning. ​Clinical Psychological Science,​ ​5​(3), 457-469.
doi:10.1177/2167702617691560
Wanner, B., Vitaro, F., Tremblay, R. E., & Turecki, G. (2012). Childhood trajectories of anxiousness
and disruptiveness explain the association between early-life adversity and attempted suicide.
Psychological Medicine​, 42(11), 2373-2382. doi:10.1017/s0033291712000438

23
medRxiv preprint doi: https://doi.org/10.1101/19010413. The copyright holder for this preprint (which was not peer-reviewed) is the
author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
It is made available under a CC-BY-NC-ND 4.0 International license .

Yang, B., & Clum, G. A. (1994). Life Stress, Social Support, and Problem-Solving Skills Predictive of
Depressive Symptoms, Hopelessness, and Suicide Ideation in an Asian Student Population:
A Test of a Model. Suicide and Life-Threatening Behavior; New York, 24(2), 127–139.
https://doi.org/10.1111/j.1943-278X.1994.tb00797.x

24