Abstract
Higher-order modulation schemes in optical communication systems that suffer from several impairments can use artificial intelligence (AI) algorithms, among other possible techniques, to mitigate these issues. In this paper, several techniques for optical communication systems have been proposed to enhance the performance of dual-polarization (DP) M-ary Quadrature Amplitude Modulation (M-QAM) as DP-16-QAM, DP-64-QAM, DP-128-QAM, and DP-256-QAM with 240Gbps data rate. Artificial neural networks (ANNs) with seven different training algorithms have been applied to optimize the optical communication system. A high optimization of modulation format identification (MFI) with accuracy up to 100% was obtained at about 13 dB OSNR and at 22 dB OSNR for the DP-265-QAM format.