MSc.

AI Dissertation - Initial Project Proposal

Artificial Intelligence (AI) has historically been viewed through two distinct lenses: symbolic AI and
stochastic AI. The former is characterised by methodologies associated with the explicit
representation of facts and knowledge, epitomised by logical reasoning. Stochastic AI, on the other
hand, encompasses techniques that encapsulate problems and corresponding solutions in probabilistic
terms, where learning emerges from pattern recognition and exposure to ever increasing amounts of
data. Though deep learning has heralded a surge in the development of stochastic AI, neither symbolic
AI nor stochastic AI individually possess the full capacity to represent, understand, and reason across
the diverse complexities present in real-world problems. Symbolic AI excels in logic and rule-based
tasks but struggles with ambiguities inherent in real-world data. Conversely, stochastic AI can discern
patterns in data but offer little transparency into the underlying decision-making processes.

Problem Description

My initial research proposal proposes an investigation into the integration of symbolic and stochastic
AI, specifically focusing on the symbiosis of Knowledge Graphs (KGs) and Large Language Models
(LLMs). Given the opportunities possible with KG’s (Peng et al. 2023), I am interested in evaluating
the extent to which the structured knowledge representation of KGs, encapsulating both explicit and
implicit facts, can enhance the reasoning capabilities of LLMs, producing a system that exhibits some
of the beneficial characteristics of both symbolic AI and stochastic AI (Ilkou & Koutraki, 2020).

The problem my research project seeks to solve is: "How can we effectively integrate explicit and
implicit structured knowledge in KGs with the learning capabilities of LLMs, and what impact
does this integration have on the model's performance and interpretability for ‘question and
answering’ and ‘knowledge exploration’?" This question arises from the observation that while
KGs can provide a structured representation of explicit facts and potentially inferred knowledge, the
mechanism for effectively exploiting this wealth of structured information in stochastic models like
LLMs is not fully explored. The challenge is therefore not only about the varying integration
strategies of knowledge graphs (Galkin et al. 2016), but also how to maintain and utilise the
distinction between explicit and implicit knowledge for knowledge understanding, and how the
integration impacts the model's ability to reason, learn, and explain its outputs.

Although symbolic AI is primarily engaged in the management of overt factual data and established
relational paradigms, the intricacies of reality frequently introduce a level of complexity that escalates
non-linearly as the quantity of these facts and rules proliferate. This invariably makes the
representation of all potential facts and rules an exponentially arduous task. It introduces uncertainty
about what knowledge is represented in the system and how accurately it reflects the real world. It is
therefore important to evaluate three main areas:

1. Uncertainty in knowledge representation: It is vital to assess the extent to which the

2. Uncertainty in Inference: Here, the aim would be to understand the system’s capacity to
draw plausible inferences from any represented knowledge. This involves evaluating the
ability of the system to correctly and coherently generate new facts based on the embedded
rules. Metrics such as precision and recall, along with other measures like logical consistency,
can be employed.
 3. Uncertainty Propagation: The system should also be assessed for how uncertainty
propagates through the chain of inferences. For instance, if a fact is inferred based on a chain
of several other facts and rules, and there is uncertainty associated with each of them, to what
extent does this affect the certainty of the final inference?

This problem, once addressed, could lead to the development of AI systems that leverage the strengths
of both KGs and LLMs. Such systems could potentially exhibit improved reasoning, generalisation,
and explanation capabilities, addressing some of the key limitations seen in both symbolic and
stochastic AI systems (Garnelo & Shanahan 2019).

Future Work

In case the problem solution is achieved earlier than expected, there are multiple avenues worth
pursuing for further insights and developments:

Scalability and Efficiency: Neuro-symbolic systems, when integrating knowledge graphs with LLMs
and applying rule embedding, may become very complex and computationally demanding. It would
be insightful to investigate techniques to optimize the computational efficiency and scalability of such
systems, while reducing the cost to train models.

Development of Explainable AI Systems: With the complexity of Neuro-Symbolic systems and their
inferred outputs, it becomes crucial to develop methods for explaining the system's reasoning process,
enhancing transparency, and aiding in its deployment in risk-sensitive settings. This is especially
important to bridge the gap between academic insights and application of novel approaches in
industry while adhering to regulatory laws and compliances. For example, a neuro-symbolic system
integrating an LLM with a knowledge graph could provide a framework to ensure regulatory
compliance, by reasoning on a set of explicitly encoded rules. Research could focus on how to best
implement this while maintaining the model's performance.

Knowledge Graph Update Mechanisms: As the world is inherently changing and the interpretation
of knowledge and information can change, it is important to study methods to automate the updating
of the KG and consider ways in which the interpretation of knowledge can change over time. This
includes both explicit knowledge (new facts) and implicit knowledge (new inferences and rules
derived from existing facts). The effective updating of the KG can ensure the system stays fresh with
the latest knowledge and potentially novel ways to interpret knowledge. (Zhong et al, 2023)

Interactive Learning and Feedback Loops: In many practical applications, models do not operate in
isolation but are part of a feedback loop, where they interact with users and learn from these
interactions. Research could focus on how to incorporate this feedback into the neuro-symbolic
framework, allowing the model to continuously learn and update its knowledge while maintaining the
models performance.

Project Objectives

Literature review: The literature review will serve a multi-faceted role within the thesis, providing a
rigorous and critical examination of the current knowledge landscape that directly relates to my
chosen topic of LLM’s and KG’s towards assimilating, evaluating, and contextualising the corpus of
academic works on subdisciplines related to the problem.

Solution Design and Curating data: Here I aim to critically evaluate and finalise both the end-to-end
design I aim to build and the data that I will be using for the thesis.
Initial Coding Section: Here the aim is to build the finalised design and showcase the reduction in
hallucinations through knowledge graphs and to compare them to a baseline LLM and LLM’s of
varying sizes/designs/types without knowledge graphs. (Aim to leverage different approaches to more
effective knowledge representations and inferencing to evaluate the extent to which different types of
knowledge graphs have different impacts on reducing hallucinations).

Post Coding Section: Here the aim is to ensure the coding section is complete, tested and evaluate
the effectiveness of each approach and using specific metrics to measure the quality and reliability of
its outputs.

Second Coding Section: Assuming the initial coding session and target is met, the next aim would be
to further extend the project by approaching one of the topics noted above in the ‘future work’ section,
and meetings with my thesis advisor to better understand the best approach to take if I reach this
stage.

Deployment: Once the coding and testing is completed, I aim to host the project online as a simple
full stack project for people to interact with. This can be in the form of a simple application hosted on
the cloud, along with a write up detailing my findings.

The last objective is to not only develop more efficient and in-depth research skills throughout the
thesis, but to also develop the necessary skills to pursue a doctorate one day and am hoping to learn
this along the way from my thesis advisor if possible.

Project Plan

Above is a Gantt chart I have created to roughly estimate the amount of time allocated to each task. I
have decided not to develop a week-by-week breakdown of each task as there are too many variables
and potential sources of errors or risks so early on in this research proposal to consider making a
rigorous plan. However, I have allocated sufficient time to the implementation of the solution in case
of any alterations to the project proposal or interesting deviations that may occur during the literature
review and interactions with my thesis advisor that could result in more time needed for coding and
evaluating my implemented solution. I have also allocated extra time to consider a topic in the ‘future
works’ section if possible and time to deploy the solution online. The aim is to complete the entire
thesis within 7 months, including deployment and completing the final write up, with one month of
contingency time.

Risks

As with any research, there are many risks and uncertainties to take into consideration and approaches
to mitigate them. Below are possible solutions to approaching certain risks that could arise:
 Risk How to approach them
Costs: As training LLM’s are both costly and There are many open-source LLM’s on
time intensive, it may be necessary to find an huggingface and online that are available for
alternative approach instead of building the research purposes; thus, it might be necessary to
LLM’s myself. spend time deciding on the criteria needed to
pick the optimal ones for this project.
I could also continue searching for cheaper
remote resources such as
https://lambdalabs.com to keep costs down and
allow for more faster training of LLM’s if I can
train them simultaneously.
Large scope: The ambit of the research A potential solution involves the employment of
proposal presents concerns due to its extensive structured research design methodologies, a
breadth. Given the accelerated pace of emergent rigorous framework, based on clearly defined
research in LLM’s, there is a chance for objectives, key research questions, and
inadvertent immersion into intricate detail and hypotheses, may help delineate the boundaries
falling behind. of the investigation; this is something I am
aiming to work on as I develop my final
research proposal and interact with my thesis
advisor during the thesis
Time Management: Thesis and dissertation I aim to improve my time management skills
work can be time-consuming and, without and implementing effective time management
proper management, can result in missed strategies like using project management tools,
deadlines. There is an impact on time setting realistic milestones, and regular progress
management and student academic achievement. tracking can help keep your work on schedule.
(Nasrullah & Khan 2015)
Dataset Curation and collection: Collecting I aim to consider multiple data collection
original data for research can be fraught with methods, ensuring ethical considerations are
potential issues, from gaining access to the met and critically evaluating my data sources. I
necessary resources to the reliability and validity could use datasets and data sources commonly
of the data I am collecting, along with different used in industry and previous papers that could
types of data such as geospatial data. make comparisons to prior works easier.

References

Expert.ai Team, 2022. Implicit v Explicit Knowledge for Language Understanding [Online]. Available from:
https://www.expert.ai/blog/implicit-vs-explicit-knowledge-for-language-understanding/ [Accessed 20 July
2023]

Garnelo, Marta & Shanahan, Murray. (2019). Reconciling deep learning with symbolic artificial intelligence:
representing objects and relations. Current Opinion in Behavioral Sciences. 29. 17-23.
10.1016/j.cobeha.2018.12.010.

Ilkou, Eleni & Koutraki, Maria. (2020). Symbolic Vs Sub-symbolic AI Methods: Friends or Enemies?.
10.1145/3340531.3414072.

Nasrullah, Shazia & Saqib Khan, Muhammad. (2015). The Impact of Time Management on the Students'
Academic Achievements. 11.

Peng, C., Xia, F., Naseriparsa, M. et al. Knowledge Graphs: Opportunities and Challenges. Artif Intell
Rev (2023). https://doi.org/10.1007/s10462-023-10465-9

Zhong, Lingfeng & Wu, Jia & Li, Qian & Peng, Hao & Wu, Xindong. (2023). A Comprehensive Survey on
Automatic Knowledge Graph Construction. 10.48550/arXiv.2302.05019.