Circumventing thread safety in rust with mutable pointers

I have a performance critical section of my rust code which I currently try to get faster with unsafe code. In this approach I grantee thread safety manually with barriers and separated index ranges for all threads in the vectors, so I do not want to compiler to check thread safety.

#[derive(Clone, Copy)]
struct CgPtrSet {
    a: *const SparseMatrix,
    b: *const DenseVector,
    x: *mut DenseVector,
    d: *mut DenseVector,
    r: *mut DenseVector,
    z: *mut DenseVector,
    accuracy: *mut f64,
    iterations: *mut usize,
}

fn cg_barrier_thread_worker(&self, ptrs: CgPtrSet, start: usize, end: usize) {
    unsafe {
        ...
        }
}

This works well while done in one thread:

self.cg_barrier_thread_worker(ptrs, start, end);

Now I run it in its own thread:

let thread_handle = thread::spawn(|| {
    self.cg_barrier_thread_worker(ptrs, start, end);
});
thread_handle.join().unwrap();

That get me the compiler output:

error[E0277]: `*const sparse_matrix::SparseMatrix` cannot be shared between threads safely
error[E0277]: `*const dense_vector::DenseVector` cannot be shared between threads safely

...and so on for all members of CgPrtSet

I have tried to solve the issue using UnsafeCell with no success. How can I “convince” the compiler that I know my use of the object is thread save?

// lets make minimal example of struct holding a pointer
struct HoldsRawPtr {
    ptr: *const u8,
}

// By implementing the Send trait, we tell compiler that this type can be transferred between
// threads without violating safety. Whether it actually is safe is our responsibility
unsafe impl Send for HoldsRawPtr {}
// Sync trait tells compiler that `&T` is safe to share between threads
unsafe impl Sync for HoldsRawPtr {}

fn main() {
    let ptr = HoldsRawPtr {
        ptr: std::ptr::null(),
    };

    // This is now possible
    std::thread::spawn(move || {
        // do something with ptr
        let _moved_ptr = ptr;
    });

    // here you can see signature of std::thread::spawn
    pub fn spawn<F, T>(f: F) -> std::thread::JoinHandle<T>
    where
        // send bound on closure requires that all captures are send
        // so we don't need Sync bound on HoldsRawPtr, though if you wrapped it in Arc,
        // you would need Sync bound
        F: FnOnce() -> T + Send + 'static,
        T: Send + 'static,
    {
        unreachable!("...")
    }
}