This research introduces the optical holographic encryption system, utilizing a novel method that significantly enhances security in underwater communications. The holographic message is encrypted using a particular algorithm called expanded Diffie–Hellman (EDH). Two methods of expansion are accomplished by the elliptic curves EDH-ec and three-dimension of a quaternion function EDH-q, respectively. Furthermore, the evaluation of security is finished using the Pollard’s Rho method for the proposed EDH algorithm. Finally, both methods have been subject to security assessments through the application of NIST tests, which evaluate the randomness of the transmitted messages. The NIST test results show that the p-value of EDH-ec is consistently higher than those of EDH-q and DH. Therefore, the obtained results demonstrated that the information was encrypted via an EDH-ec algorithm, which had better security than EDH-q under unsecure acoustic channel when face the unpredictability and complexity condition.