I figured out exactly what the problem is (not the solution though):
I’m using “vh” unit for a lot of my vertical spacing in the page. What happen is that when you scroll up and down in chrome for android, it makes the address bar on the top hide or show, and this causes the viewport height to change, making my vertical spacings resize. But for some reason the spacings are resized only after the scroll animation ends or the user releases their finger from the screen.