Introduction

Quantum spin liquids (QSLs) are exotic Mott insulating phases without long-range magnetic order, but having long-ranged entanglements and fractionalized excitations1,2,3,4,5. The Kitaev honeycomb model, which hosts accurate ground state and Majorana excitations6,7,8, attracted lots of interest in the realization of QSLs. As a Kitaev QSL candidate9,10, the Mott insulator (MI) α-RuCl3 (as shown in Fig. 1a) has been extensively investigated in the past decade11,12,13,14. It has been shown that α-RuCl3 is proximate to the Kitaev paramagnet between 7 and 120 K15,16,17, where a continuum was observed in the spectrum of inelastic neutron scattering (INS)15,18,19,20,21, Raman scattering16,22,23,24,25,26 and other spectral techniques27,28. Furthermore, under external magnetic fields, thermal transport measurements have detected half-quantization of the thermal Hall conductance29,30,31,32 as well as a quantum oscillation in the longitudinal thermal conductance33,34 despite the anomalous phonon contribution35,36. These observations indicate that α-RuCl3 is an ideal platform to study fractionalized spin excitations.

Fig. 1: Unconventional Mott transition in α-RuCl3 in proximity to graphite.
figure 1

a Crystal structure of α-RuCl3 in the ab and bc planes; b STM morphology of α-RuCl3 flakes transferred on a graphite substrate; c Selected dI/dV spectra taken on α-RuCl3 flakes with thicknesses ranging from 1-ML to 5-MLs showing two types of line-shapes, one with a large gap and the other with a reduced gap (Vbias: 0.5–1 V, Iset: 0.5–1 nA); d Left: averaged dI/dV spectra taken on a series of α-RuCl3 thin flakes with different thicknesses. Right: Plots of the relationship between the gap-amplitude (G) and the appropriate thickness (T) expected from the logarithmic function \(G=\alpha -\beta \ln (T+\gamma )\) (where α, β, and γ are constants).

Full size image

Aside from the magnetism, the charge properties of Kitaev materials have attracted increasing interest37,38,39,40. Nevertheless, a significant concern is that these materials, like α-RuCl3, are electrically inert MIs, which prevent any low-energy charge excitations. Forming a heterostructure with a metallic system has been demonstrated to be an excellent strategy for injecting active charges and triggering exotic phenomena. For instance, it was proposed that electrons tunneled from the substrate can locally probe and control the magnetic fractional excitations in Kitaev thin film41,42,43,44,45. Investigations of charge transfer and magnetic phase transition at the interface between α-RuCl3 and graphene46,47,48,49,50,51 or MnPc52 were reported. Potentially, the coupling between the charge, spin, and lattice degrees of freedom are helpful to tune the magnetic interactions and to manipulate the neutral fractional excitations, as proposed in a series of theoretical papers53,54,55,56,57,58. Therefore, it is an important issue how the Mott nature can be modified in Kitaev thin film on metallic surface.

Our early works have revealed an unconventional Mott transition in few-layer α-RuCl359,60. Here, we report our new scanning tunneling microscopy/spectroscopy (STM/STS) experimental observations of incommensurate super-modulation of both charge density and lattice morphology on α-RuCl3 during its transition from MI to charge transfer insulator (CTI). Our results suggest that the CTI of few-layer α-RuCl3, with a significantly reduced charge gap, enables low-energy charge excitations which are strongly coupled to the spin and lattice degrees of freedom. Specifically, the anti-ferroelectric dipole order in few-layered α-RuCl3 has induced the unconventional charge order under intrinsic fluctuation here. Our work not only reveals the charge properties of the Kitaev material α-RuCl3, but also sheds light on the understanding of charge-spin coupling in general quantum magnets.

Results

We perform the STM/STS study of a few layers α-RuCl3 that have been transferred onto the surface of graphite. A selected STM topography recorded at 77 K is shown in Fig. 1b. The spectra were systematically acquired on a series of thin flakes ranging from 1 to 5-MLs, as well as thicker flakes (Supplementary Fig. 1), where the thicknesses were measured through the STM cross-sectional profiles. Two types of representative dI/dV spectra, both of which exhibit a full charge gap, are shown in Fig. 1c. The first type, which belongs to the flakes of 4-MLs and above, bears resemblance to that of bulk (Supplementary Fig. 1 and refs. 61,62), i.e., the gap values remain close to 2 eV, revealing a strong insulating state. The second type is identified in a few layers that are situated in close proximity to graphite and exhibit a dramatically reduced charge gap, which will be later referred to as the reduced gap (RG). During the transition between the two gapped phases as decreasing the thickness, the dI/dV spectrum of MI experiences a dramatic transfer of spectral weight from Hubbard bands (HBs) to the sides adjacent to the Fermi level (FL) (Supplementary Fig. 2). dI/dV spectra pertaining to the RG phase were collected on a few dozens of thin flakes (see left in Fig. 1d), and we found a logarithmic decrease in the gap as the surface approached the substrate, as depicted in right panel of Fig. 1d.

In order to further understand the electronic states of RG phase in α-RuCl3, we first focus on 2-MLs. The energy-dependent dI/dV maps (Fig. 2a–h) were performed on an area exhibiting atomic-resolved STM morphology on α-RuCl3 surface, as shown in Fig. 2i. The most important feature is the super-modulation of the surface local density of state (LDOS), which resembles a charge density wave order. The modulation is also observable on the lattice (see Fig. 2i). The findings presented in Supplementary Fig. 3 reveal that the super-modulation primarily occurs in the occupied states (negative sample biases). As depicted in Fig. 2e–g, it is most pronounced within the sample biases of −200 mV and −800 mV, and diminishes as the bias deviates from this range (refer to Supplementary Fig. 3c for further details). The shaded circles in Fig. 2c, e demonstrate a contrast reversal occurring at the band-edges. Furthermore, it should be noted that while the amplitude of the super-modulation exhibits a strong correlation with the biases, the patterns remain relatively static with respect to the energy.

Fig. 2: Incommensurate charge order in α-RuCl3.
figure 2

a–h show sample bias-resolved dI/dV maps measured at Vbias = 500 mV, Iset = 1 nA on a 2-ML α-RuCl3. Color dots in (c) and (e) indicate the reversal patterns with weaker contrast at ±200 mV, i Atomic-resolved STM image recorded on the same place in (a–h) shows that the super-modulation is also observable on the lattice (Vbias = 500 mV, Iset = 1 nA); j FFT of the STM image shows the Bragg peaks of the α-RuCl3 lattice (white and red circles) and peaks of the static super-modulation inside the 1st BZ (green dashed hexagon). kRu and kCl denote the wave vectors of ruthenium (Ru) and chlorine (Cl) lattice, respectively. ksm is the wave vector of the super-modulation. k The line cut along the Γ-K direction shows that ksm is ~0.39 r.l.u. (r.l.u. denotes reciprocal lattice unit of α-RuCl3 2π/a, a = 6 Å).

Full size image

To further analyze the super-modulation, a fast Fourier transform (FFT) was performed. The Bragg peaks corresponding to both ruthenium (Ru) and chlorine (Cl) atoms are highlighted by the white and red circles in Fig. 2j. The set of peaks observed along Γ-K direction within the first Brillouin zone (BZ) is clear evidence of the static super-modulation present on the morphology. Upon a line cut along the white dashed line in Fig. 2j, it was determined that the wave vector ksm has a value of approximately 0.39 r.l.u (r.l.u. denotes reciprocal lattice unit of α-RuCl3 2π/a, a = 6 Å) (Fig. 2k), indicating that the super-modulation is incommensurate.

Our systematic investigation confirms that such surface modulations are commonly occurring in 2-MLs. The wave vector ksm remains static and does not depend on the orientations between α-RuCl3 and graphite in different samples. In a second 2-MLs sample, the bias-dependent STM images exhibit a conventional α-RuCl3 lattice with moderate morphological modulation under positive biases (Supplementary Fig. 4a), similar to that shown in Fig. 2i. In contrast, the super-modulation arises and prevails throughout the surface under negative biases, which becomes particularly prominent around a bias of −600 mV, as shown in Supplementary Fig. 4a. Furthermore, we find the modulation on surface LDOS and around defects evolve almost synchronously with the morphological super-modulation under variation of biases, as shown in Supplementary Fig. 4b. The evolution is also revealed in the bias-resolved Fourier components of the dI/dV data (Supplementary Fig. 4c), where the FFT patterns of the super-modulation become more prominent and evolve from point-like pattern to nearly ring-shape around the Γ point as bias shifts towards the negative end (Supplementary Fig. 4c).

We found that the super-modulation on the lattice is absent for samples of 4-MLs and thicker. We took dI/dV map for a thicker (Supplementary Fig. 1) and a 4-MLs samples (Supplementary Fig. 5), respectively. No super-modulations were observed in the surface DOS, and the data are indistinguishable from the orbital textures. Subsequently, the surface morphology and LDOS were measured on a lower terrace (3-MLs), wherein the RG phase appeared but without any super-modulation. The bias-dependent dI/dV maps are illustrated in Fig. 3a. The images exhibit orbital textures linked to the Ru t2g bands at the biases surpassing the UHB (240 and 300 mV). Within the sample biases of −300 to −480 mV, the textures in occupied states display a ring-shaped pattern sharing the symmetry of the Cl arrangements (represented by green dots). At bias below −600 mV, the orbital textures converge towards the center of the ring and become blurred. The ring-shape pattern can be attributed to the 3p orbitals, as shown in the inset of Fig. 3b. Quantitative analysis of the spectra obtained from the Ru and Cl sites reveals that the ring-shape texture of Cl 3p orbitals corresponds to the shoulder peak in the dI/dV spectra, as depicted in Fig. 3b.

Fig. 3: Mott insulator (MI) to charge-transfer insulator (CTI) transition.
figure 3

a Bias-resolved dI/dV maps taken on the same region of 3-MLs α-RuCl3 (Vbias = 900 mV, Iset = 1 nA). Orbital textures corresponding to the Ru sites (light blue dots) are detected at the edge of UHB (180–300 mV). In contrast, the negative bias side displays the textures of Cl 3p orbitals (green dots). b A comparison of averaged dI/dV spectra taken on the Ru and Cl sites reveals enhanced occupied states and distinct dip-humps at Cl sites (Vbias = 800 mV, Iset = 800 pA). Inset schematically shows the configuration and pattern of Cl 3p orbitals. c Schematic DOS (density of states) from MI to CTI when the Cl 3p orbitals enter the Mott gap of Ru 4d electrons with changing thickness (layer number). U(L)HB is an abbreviation of upper (lower) Hubbard band; U is the Coulomb interaction and △ denotes the charge transfer gap.

Full size image

The experimental detection of the Cl 3p orbitals is crucial for understanding the reduction of Mott gap. First, notice that part of the electrons in graphite are transferred towards the α-RuCl347,48,49,50,63. If these electrons are doped into the Mott-HBs of α-RuCl3, they will induce a finite DOS at zero bias (FL) by causing double occupancies and charge carriers (i.e. the doublons). Nevertheless, a (reduced but finite) charge gap was persistently observed, which leads to the conclusion that the electrons do not enter the Mott-HBs; rather, they formed an interfacial dipole layer as a result of the difference in chemical potential between α-RuCl3 and graphite51. The dipole layer will significantly change the energy levels of the ions, especially the outer shell 3p orbitals of anions which are less screened (in contrast, the energy levels of the 4d orbitals of the cations are less affected due to screening of the outer shell orbitals)47,64. Apparently, for 2-MLs and 3-MLs, the energy levels of the Cl 3p orbitals are pushed into the Mott gap between 4d5 and 4d6 of the Ru3+, which turns the flake of α-RuCl3 into a CTI with dramatic RG, as schematically shown in Fig. 3c. The transition from MI to CTI is depicted in the form of additional low-energy states on Cl sites (Fig. 3b), as well as the noticeable shift in band-edges toward FL as the surface approached the heterointerface (Fig. 1d). On the other hand, in contrast to the situation depicted in Fig. 3, we did not observe any orbital textures originating from Cl 3p in the dI/dV maps of 1-ML case, even though there are in-gap states in the dI/dV spectrum60. Here, we propose that the enhanced crystal field due to the possible buckling of Ru atoms in 1-ML65 and the strong electric field by the charge transferred from graphite to the surface47 restructure the orbital configurations in α-RuCl366, keeping 1-ML a MI. For 4-MLs and above, the electric field reduces quickly such that the energy level of Cl 3p orbital is not close to the FL, hence the charge gap is not significantly reduced.

After the finish of the current work, a preprint67 reported the observation of quantum oscillations at 4.2 K on 1-ML α-RuCl3 that was directly grown on graphite. The sample in ref. 67 exhibits a reduced Mott gap of around 0.6 eV, and the STM images also reveal the super-modulation patterns extending beyond the oscillation around the defects, as discovered in our 2-ML samples. Noticing that the growth 1-ML α-RuCl3 on graphite is thicker than the transferred 1-ML sample, this may be the reason that super-modulation was not observed in our 1-ML sample.

Discussion

The charge super-modulations in electron Fermi liquids, which have a large electron FS, are often linked to CDW, moiré patterns, or quasiparticle interference (QPI). In the case of the few-layer α-RuCl3 on graphite where a full charge gap is present, these mechanisms can all be ruled out. The Peierls-type CDW and correlation-driven charge order, if it takes place, would appear at considerably lower energy68. In addition, the formation of moiré patterns is implausible due to the significant lattice mismatch between α-RuCl3 and graphite47, as is already confirmed by the absence of the super-modulation in 1-ML case in our recent experiments59,60. The observation of comparable super-modulation on 1-ML α-RuCl3 growth directly on graphite in ref. 67 can further effectively exclude the moiré patterns formed due to a twist angle between two α-RuCl3 layers. Furthermore, due to the lacking of bias dependence, the observed super-modulation is inconsistent with the QPI patterns as well69. Finally, with charge transferred from graphite to heterointerface, the nesting between two hole-pockets at the positions of Dirac cones (K and \({K}^{{\prime} }\)) with wave vector k may affect the charge distribution in layered α-RuCl3. However, this can’t interpret the fact that wave vector of the charge super-modulation in 2-MLs α-RuCl3 is independent on the relative orientation of the substrate.

A plausible interpretation of the super-modulation is from the interference of Majorana spinons. At temperatures slightly above the Néel temperature Tc, the material α-RuCl3 in 2D limit is proximate to a Kitaev spin liquid owing to the combined thermal and quantum fluctuations. Especially, in a temperature region the thermally excited Z2 fluxes yield a finite Fermi surface of the Majorana spinons, which are called thermal Majorana metal in literature70,71,72,73,74,75. The nesting of the Majorana Fermi surface can generate spinon density waves. The internal coupling between spinons and chargons (i.e., holons or doublons depending on the bias voltage) will potentially cause a charge order and quantum oscillation of the electron density76. However, the size of Majorana fermi surface should be temperature dependent, which is inconsistent with the fact that the super-modulations are detectable at both temperatures of 77 K and 4.2 K with the same wave vector67,77. At this point, there exists no clear evidence to establish a connection between the observed super-modulation and Majorana fermions.

Here we propose an alternative mechanism that is closely related to the antiferroelectricity of α-RuCl3 caused by the charge hopping between anions and cations in the CTI in an inhomogeneous manner78,79. At zero electric field, the electric dipole moments are oriented anti-ferroelectrically along the c-direction. However, the pattern of dipole order can be changed by the strong internal electric fields from the charges transferred to the interface or at vacancy defects. To investigate the effect of electric field, we consider an effective model of the electric dipoles (Supplementary note 4) containing dipole-dipole interaction H1 with strength Jd, anisotropic interaction \({H}_{2}={\sum }_{\langle i,j\rangle \in (\alpha \beta )\gamma }[K{p}_{i}^{\gamma }{p}_{j}^{\gamma }+\Gamma ({p}_{i}^{\alpha }{p}_{j}^{\beta }+{p}_{i}^{\beta }{p}_{j}^{\alpha })]\) resulting from charge-spin coupling, easy-axial anisotropy and internal electric field \({H}_{3}={\sum }_{i}[D{({{{\bf{p}}}}_{i}\cdot \hat{c})}^{2}-{E}_{i}{{{\bf{p}}}}_{i}\cdot \hat{c}]\), assuming that the Fourier component of the electric field obey Gaussian distribution in momentum space \({E}_{k}={E}_{0}{e}^{-\frac{{k}^{2}}{{\alpha }^{2}}}\) (with α = 0.02 plus an anisotropy). Adopting the parameters D/Jd = −1.28, K/Jd = 0.1, Γ/Jd = −0.4, we obtain the phase diagram shown in Fig. 4a. Due to the easy-axial anisotropy potential D and the dipole-dipole interactions Jd, the dipoles favor an anti-ferroelectric ground state (Fig. 4b) when the electric field E0 is weak E0/Jd < 2.59. When the electric field is strong E0/Jd > 2.72, the ground state will be turned into a polarized state (Fig. 4c). However, when the electric field falls in an intermediate window 2.59 < E0/Jd < 2.72, the dipoles form incommensurate order with the wave vector k ~0.40 r.l.u. (see Fig. 4d) which is very close to ksm=0.39 r.l.u. of the experimentally observed super-modulations. The incommensurate dipole order p(r) induces a CDW order ρ(r) = ∇ ⋅ p(r) with the same wave vector k (as shown in Fig. 4e). This provides a possible origin of the super-modulations in the STM experiment. In the experiment, the electric field E0 is generated by the interfacial dipole between α-RuCl3 and graphite, and it decays on α-RuCl3 surface as the thickness increases. Consequently, the intralayer dipoles of α-RuCl3 form the incommensurate order when the electric field falls in the certain window, as evidenced by 2-ML sample. We infer that E0 on 3-ML case has been outside that range with E0/Jd < 2.59, and the intralayer dipoles favor an anti-ferroelectric ground state. The incommensurate dipole order may also contain the contribution of lattice distortion due to softening of certain phonon mode caused by electric force between the substrate and the α-RuCl3 layer35,80.

Fig. 4: Antiferroelectricity and the incommensurate dipole order.
figure 4

a Phase diagram of the dipole model, where ‘AFE’ stands for the anti-ferroelectric phase. Cartoon picture of the dipole distribution in three phases: b the anti-ferroelectric phase (the blue/red arrows stand for A/B-sublattice respectively); c the polarized trivial phase; d the incommensurate dipole order phase. e Illustration of the charge density wave(CDW) order associated with the incommensurate dipole order.

Full size image

In summary, we performed a systematic STM/STS study of a few-layer α-RuCl3 proximate to a graphite surface. Our data indicate that α-RuCl3 is changed from a bulk MI to a CTI (2-MLs and 3-MLs) due to the shift of energy levels of the 3p orbitals in the Cl− anions, which guarantees the charge excitation near the FL. Furthermore, exotic incommensurate super-modulations of charge states were observed in 2-MLs samples, which is here consistently interpreted as a charge order associated with an incommensurate electric dipole order. The findings highlight the significance of anions in tuning the Mott-HBs in Kitaev materials and indicate the strong coupling between charge and spin degrees of freedom, which may result in an electrically accessible method of detecting the magnetic properties. Our work urges the experimental studies to further explore the electric dipole degrees of freedom in α-RuCl3 and to discover the microscopic mechanism of the interactions between charge and spin.

Methods

In this work, we exfoliated the α-RuCl3 thin flakes using the scotch tapes, from a bulk crystal synthesized by vacuum sublimation of commercial high purity (99.99%) α-RuCl3 powder. After repeatedly reducing the thickness of the film, we exfoliated again using a thermal release tap. Then, the thermal release tap with the α-RuCl3 thin flakes was attached to a fresh surface of the HOPG substrate. After heating the sample to 120 °C for 20 seconds in the atmosphere, the α-RuCl3 thin films were released on the HOPG surface. Then the sample was annealed at 280 °C in ultra-high vacuum chamber (1E-10 Torr) in STM system for at least 2 hours for degassing and improving the contacting quality, before STM/STS measurements. We measured α-RuCl3 flakes with different thicknesses randomly, and captured the STM topographic images and the STS spectra on the chosen regions. In this experiment, a commercial STM system (Unisoku-1300) has been used at a base temperature of 77 K. The STM tip is made by tungsten wire with self-corrosion in 4 mol/L NaOH. The STS spectra measurements were carried out using a lock-in technique at a frequency of 707 Hz, and a modulation voltage ranging from 5 mV to 10 mV.