I don't like the insertion of the word not, in either place. The expression [adverb] speaking is used often, and it's almost idiomatic. For example:
- Generally speaking, it's rainy in Seattle.
- Technically speaking, the Internet isn't down – it's your connection that's down.
- Financially speaking, driving an ice cream truck isn't a good career path.
- Ethically speaking, I had to tell the buyer that the engine is leaking oil.
This construct means, essentially, from an [adjective] point of view... So, these sentences would mean:
- From a general point of view, it's rainy in Seattle.
- From a technical standpoint, the Internet isn't down – it's your connection that's down.
- From a financial perspective, driving an ice cream truck isn't a good career path.
- As a matter of ethics, I had to tell the buyer that the engine is leaking oil.
If I wanted to somehow negate this construct, I would restructure the sentence, rather than using the word not.
- It usually rains in Seattle, but today it's sunny.
- Technically, the internet isn't down – but I know what you mean.
- You won't make a fortune driving an ice cream truck, but I can see how the job might be fun.
- I could have withheld the oil problem, but that would have been unethical.
Getting back to your example, if you are refuting the person from a perspective that's not the financial perspective, then you should pinpoint that perspective:
- Yes, at least morally speaking, your opinion sounds right, even if ...
Alternatively, if you want to emphasize that your statement is excluding the financial perspective, you could word that in a manner such as one of these:
- Yes, financial considerations aside, your opinion sounds right, even if ...
- Yes, disregarding financial matters for the moment, your opinion sounds right, even if ...
but I would strongly recommend against Yes, not financially speaking...