I think you are looking for *quantum machine learning* (QML), which is a relatively new field that sits at the intersection of quantum computing and machine learning.

If you are not familiar with quantum computing (QC) and you are interested in QML, I suggest that you follow this course by prof. Umesh Vazirani and read the book Quantum Computing for Computer Scientists (2008) by Yanofsky and Mannucci. If you have a solid knowledge of linear algebra, you should not encounter big problems while learning QC. Be prepared to deal with the weirdness and beauty of qubits, quantum entanglement, and so on.

If you want to directly dive into QML (although I am not familiar with the details of QML, I suggest that you first get familiar with the basics of QC, which I am familiar with) there are already several courses, papers, overviews, and libraries (such as TensorFlow Quantum) on quantum machine learning.

If you are interested in quantum reinforcement learning, maybe have a look at the paper Quantum Reinforcement Learning (2008) by Daoyi Dong et al.