Should I switch to chemical sunscreen to avoid white cast?

Yes, switch to chemical sunscreen because it absorbs into the skin and leaves zero white cast, unlike mineral sunscreens that sit on top and cause it. Hybrid sunscreens exist, but pure chemical is the simplest fix for most people.