Panicle development is an important determinant of the grain number in rice. A thorough characterization of the molecular mechanism underlying panicle development will lead to improved breeding of high-yielding rice varieties. Frizzy Panicle (FZP), a critical gene for panicle development, is regulated by OsBZR1 and OsARFs at the transcriptional stage. However, the translational modulation of FZP has not been reported. We reveal that the CU-rich elements (CUREs) in the 3′ UTR of the FZP mRNA are crucial for efficient FZP translation. The knockout of CUREs in the FZP 3′ UTR or the over-expression of the FZP 3′ UTR fragment containing CUREs resulted in an increase in FZP mRNA translation efficiency. Moreover, the number of secondary branches (NSB) and the grain number per panicle (GNP) decreased in the transformed rice plants. The CUREs in the 3′ UTR of FZP mRNA were verified as the targets of the polypyrimidine tract-binding proteins OsPTB1 and OsPTB2 in rice. Both OsPTB1 and OsPTB2 were highly expressed in young panicles. The knockout of OsPTB1/2 resulted in an increase in the FZP translational efficiency and a decrease in the NSB and GNP. Furthermore, the over-expression of OsPTB1/2 decreased the translation of the reporter gene fused to FZP 3′ UTR in vivo and in vitro. These results suggest that OsPTB1/2 can mediate FZP translational repression by interacting with CUREs in the 3′ UTR of FZP mRNA, leading to changes in the NSB and GNP. Accordingly, in addition to transcriptional regulation, FZP expression is also fine-tuned at the translational stage during rice panicle development.