Foundation models are subject to an ongoing heated debate, leaving open the
question of progress towards AGI and dividing the community into two camps: the
ones who see the arguably impressive results as evidence to the scaling
hypothesis, and the others who are worried about the lack of interpretability
and reasoning capabilities. By investigating to which extent causal
representations might be captured by these large scale language models, we make
a humble efforts towards resolving the ongoing philosophical conflicts.