Chart.js zoom instantly zooms in fully on first point no matter what type of zoom used

After adding zoom to my line chart no matter what I do it instantly zooms in to the first point without the option to zoom back out (outside of the reset zoom function). The zoom out function also doesnt work, zoom in function results in the same problem.

Normal
After any attempt to zoom

const pdmdata = JSON.parse(document.getElementById('div_pdmdata').dataset.chart);
const ctx = document.getElementById('canvasChart_1').getContext("2d");
const radars = {'hidden': 'hidden'};
let linechartData = {
    datasets: [{
        label: 'Constant',
        data: pdmdata[0],
        borderWidth: 1
    }]
};
for (let i = 0; i < Object.keys(pdmdata[1]).length; i++) {
    linechartData.datasets.push({
        label: radars[Object.keys(pdmdata[1])[i]],
        data: pdmdata[1][Object.keys(pdmdata[1])[i]],
        borderWidth: 1,
        tension: 0.1
    });
}
const config = {
    type: 'line',
    data: linechartData,
    options: {
        maintainAspectRatio: false,
        interaction: {
            mode: 'index',
            intersect: false,
        },
        stacked: false,
        plugins: {
            title: {
                display: true,
                text: 'Mode S'
            },
            zoom: {
                zoom: {
                    drag: {
                        enabled: true,
                        // mode: 'x',
                        threshold: 24
                    },
                    mode: 'x'
                }
            }
        },
        scales: {
            x: {
                reverse: true
            }
        },
        elements: {
            point: {
                radius: 0
            }
        }
    },
};
myChart = new Chart(ctx, config);
function resetZoom(){
    myChart.resetZoom();
}

I've tried all the zoom types; zoom, drag and pan. Even using drag in the middle of the chart makes it end up like the zoomed image above. No errors in console whatsoever. I've also tried setting the speed to 0.001 but to no avail. Weirdest thing is when I make a duplicate of this chart below it that one does zoom as intended. I have no idea why it would though.

Okay so I found my issue, chartjs-zoom does not like objects as data without also setting labels yourself. Here's an example: (Only used 1 dataset while testing to make it easier to test)<br><br>
This works:

    let linechartData = {
        label: ['one', 'two', 'three', 'four', 'five'],
        datasets: [{
            label: 'Constant',
            data: [1, 2, 3, 4, 5],
            borderWidth: 1
        }]
    };

This does not work:

    let linechartData = {
        datasets: [{
            label: 'Constant',
            data: {'one': 1, 'two': 2, 'three': 3, 'four': 4, 'five': 5},
            borderWidth: 1
        }]
    };

But this does:

    let linechartData = {
        label: ['one', 'two', 'three', 'four', 'five'],
        datasets: [{
            label: 'Constant',
            data: {'one': 1, 'two': 2, 'three': 3, 'four': 4, 'five': 5},
            borderWidth: 1
        }]
    };

So conclusion, when using chartjs-zoom always assign the labels beforehand instead of letting a dataset data object assign them automatically, which works just fine for normal charts without zoom. Never figured out why if I made a 2nd chart with the exact same config that chart did work though.