There are any number of things that could be contributing to it, but the most likely culprit is the frame rate...24 is a freakin' minefield! The biggest issue with 24fps, is that there are multiple formats: true 24, 24P, 24PsF, and 23.976 (sometimes abbreviated as 23.98). It's important to note that these are all slightly different...they are not interchangeable. Some cameras will label 23.976 as "24" or "24P," even though they aren't technically running in either of those formats (you'd need to check the manual to confirm the actual frame rate). So, that's my first guess...despite your efforts to avoid it, you've probably got a mismatch in frame rates.
Another possibility is the edit software (specifically Final Cut Pro). We've run into a particular issue here at work with FCP7 where it rewrites the header information of the audio files on import. This causes the audio to run at the incorrect rate (whether it runs faster or shorter seems to be a coin toss). The only way we've found to fix this issue (when it happens) is to delete the offending audio files from the system (remember the header file has already been rewritten, it's buggered for all time), trash all of the preferences for FCP, then re-copy and import the audio files into the project.
That's all I can think of at the moment. If you have a clap stick (or something similar to stand in for one), using it at the head and tail of a scene can at least give you an additional option for fixing during edit (time compression/expansion to get it to match).