I just realized that ADM3053 is NOT a 3.3V device. It says 5V or 3.3V on VIO, but VCC requires 5V. The schematics I have used is also wrong as it does not feed power to VIO at all. The example on the datasheet is very misleading in this case. Looking at ADM2583 that I uses for RS485 this is all sorted – it actually IS a 3.3V device. This is a bit annoying because it means I need to maintain both 3.3V and 5V since 3.3V is needed elsewhere.