Currently, there is no way to trigger a DA on Modal Cancel, only Modal Close. There is a way to code it yourself, but it would be better for it to be declarative. https://askmax.blog/2016/11/04/handling-dialog-cancel-event/
APEX 21.2 introduces a new Dialog Closed or Canceled event that will fire if a user presses the Escape key or the X button in the dialog. In addition, the JavaScript this.data.closeAction will returns if the dialog has been closed or canceled.