How to make svg image not draggable and clickable

I'm working on svg image slider and everything works fine until I load image with text. Then when I drag my mouse text gets selected and that's normal. When I block it with css properties:

  user-select: none
  -moz-user-select: none
  -webkit-user-drag: none
  -webkit-user-select: none
  -ms-user-select: none
  pointer-events: none

Everything works again but while doing that every onclick action in svg element gets blocked. My question is how can I make element draggable and enable onclick action in same time.

Edit

I'm providing sample code for my problem, I hope it will clarify everything.

<!-- begin snippet: js hide: false console: true babel: false -->

<!-- language: lang-js -->

document.getElementById('inspector-wrapper').addEventListener('mousedown', () => {
      const lastObjX = this.imgX; // getting img X,Y
      const lastObjY = this.imgY;
      this.lastX = this.currentX; // getting start X,Y
      this.lastY = this.currentY;
      console.log(this.currentX);
      this.interval = setInterval(() => { // when holding mouse1
        this.definePosition(lastObjX, lastObjY);
      }, 10);

      private definePosition(lastObjX: number, lastObjY: number) {
        const distanceToMoveX = this.currentX - this.lastX;
        if (distanceToMoveX !== 0) {
          this.imgX = lastObjX + distanceToMoveX; // setting new X
        }
        if (this.isYAvailable()) { // if Y available set new value
          const distanceToMoveY = this.currentY - this.lastY;
          if (distanceToMoveY !== 0) {
            this.imgY = lastObjY + distanceToMoveY;
          }
        }
        this.setImgXY();
      }

<!-- language: lang-css -->

#inspector-wrapper
  position: absolute
  overflow: hidden
  background-color: transparent

\:host::ng-deep object
  position: absolute
  left: 0
  top: 0
  user-select: none
  -moz-user-select: none
  -webkit-user-drag: none
  -webkit-user-select: none
  -ms-user-select: none
  pointer-events: none

<!-- language: lang-html -->

<div id="inspector-wrapper">
  <object data="../assets/main.svg" id="test" type="image/svg+xml"></object>
</div>

<!-- end snippet -->

<!-- begin snippet: js hide: false console: true babel: false -->

<!-- language: lang-js -->

document.getElementsByTagName("img")[0].onclick = function() {
  console.log("Pointer events work");
}

<!-- language: lang-css -->

img {
  user-select: none;
}

<!-- language: lang-html -->

<img src="https://www.gravatar.com/avatar/4db6ced4189e044379de4d14c0208ddf?s=48&d=identicon&r=PG&f=1" draggable=false>
<!-- I needed a valid image, so I used my profile picture -->

<!-- end snippet -->

This uses user-select: none to disable selection, draggable=false to disable drag-and-drop, and uses an onclick event to demonstrate pointer events still work.