Disclosures made by the inventor less than 1 year before the effective filing date is not prior art in the US (35 USC § 102(b)(1)). Accordingly, as long as the effective filing date (which can be the date of filing a provisional, where the benefit of the provisional is validly claimed in a subsequent non-provisional) is less than 1 year after such a disclosure, the disclosure is not prior art in the US for that inventor. It is, however, prior art for everyone else in the US.
However, it is still prior art in most other jurisdictions for the inventor. Notably, there are no such general grace periods in Europe or China. The disclosure would therefore prevent you getting a patent there.
Can someone still knock your patent out, if they can prove they had and worked on the idea longer?
No. Secretly working on something is not prior art, no matter how long they have done so. Prior art must be available to the public (35 USC § 102(a)).
This may, if the quite strict requirements are met, give rise to prior user rights (35 USC § 273), which can restrict the ability of a patent holder suing the prior user. However, this would not invalidate the patent.