Last year I stayed in an area known as Upper-Grandview. Beautiful views of the Hudson but not cheap.
Try Nyack, NY, a little more downmarket/much cheaper (still a nice area, from what I saw on my bike), around 20km’s or so outside of Manhattan.
Connected by both bus/rail routes to Manhattan & around 30-60mins travel depending on time/rush hour.