Ginkgo biloba L. seed (GBS) is a traditional Chinese medicinal and food resource, and its use as a raw material to brew fruit vinegar can provide a new direction for the utilization of abundant GBS resources. However, balancing ginkgotoxin degradation, active ingredient retention, and bitter flavor attenuation represents a bottleneck in the ginkgo vinegar brewing process. In this study, six strains of Aspergillus oryzae were isolated from naturally fermented soybean paste blocks and identified using multiphase taxonomy methods. Among them, strain A. oryzae SSC3-6 had the fastest growth rate and the best coverage effect for preparing ginkgo koji, and its amylase, glycosylase, and cellulase activities were all the highest, with values of 8.35, 4.17, and 7.89 U/mL, respectively. When comparing enzymatic hydrolysis rates, ginkgo vinegar produced using A. oryzae SSC3-6 has a faster production rate, and the MPN content in the vinegar decreased significantly from 84.65 to 24.30 μg/mL (P<0.05). The contents of γ-aminobutyric acid and flavonoids increased significantly from 0.15 mg/mL and 83.00 mg/L to 1.08 mg/mL and 133.96 mg/L, respectively (P<0.05). Furthermore, the A. oryzae ginkgo vinegar exhibited obvious aromas of raw GBS, with a sweet aftertaste. Moreover, the composition of volatile components was more diverse and balanced, with a lower abundance. Therefore, the use of A. oryzae in the brewing of ginkgo vinegar can effectively control MPN content, maintain or even increase the content of active substances, and improve the unique bitter flavor of ginkgo fruit. The results of this study represent an important reference for the high-quality and non-toxic brewing of ginkgo vinegar in China.