This may sound strange, but the best way to find the count is often to follow the main melody, the lead vocals if there are any.
To get to grips with this song my suggestion would be: listen to the song from the beginning and - in your mind - sing along using only the sentence "in a mellow tone 2 3 4 5 in a mellow tone 2 3 4 5" and keep repeating it throughout the song, even right through timestamp 3:15 and the saxophone solo. The one count will stay on the word "tone" in the sentence "in a mellow tone". (To be exact: "tone" comes just a little bit before the first count, but it feels like it is part of the beat, of a movement that drops down on count one). The word "in" is NOT on count one but on count six!
The next step is to start listening from timestamp 3:15 onwards and see if you can pick it up, singing along in your head. Go on untill Ella comes back in and hear if you were right. Keep trying over again. Maybe you will notice that the saxophone also plays "sentences" that often end on the first count or have "syllables" that are emphasized on the first count. (Of course this is not the case in every song, but it applies to this one and to many others).
These things are hard to explain but easy once you get the feeling. The danger of trying to understand it by listening to the drummer or the bass player is that you are focusing too much on the details. On the other hand: this is jazz, everybody is playing around with the rhythm, the drummer in this song is the one that is most steady.