Currently, using Delete to dissolve a selected point causes it to change shape:

The default deletion mode for dissolving a point should aim to rescale the adjacent handles to best approximate the original curvature. See the algorithm described at https://math.stackexchange.com/a/879213/94039 which will hopefully work decently even in cases where an approximation of the original is required.