Tries from Jone Naikabula and debutant Amato Fakatava in the first half and one from Semisi Masiwera in the second kept Japan in front for most of the game, but Tonga scored two tries of their own and were pressing for another until the end.
Winger Naikabula opened Japan's account off an attacking scrum but just three minutes later a smart pickup by Tongan blindside Vaea Fifita around a weakly defended ruck set up scrumhalf Sonatane Takulua to run one in for the visitors.
The back and forth continued throughout, punctuated by a series of handling errors on both sides which left little room to build phases and pressure.
With less than two minutes until the final whistle, Japan led by just five when an interception by Tonga outside centre Afusipa Taumoepeau almost resulted in a try that would have levelled up the scores.
A last ditch try-saving tackle by replacement fullback Kotaro Matsushima, however, saved Japan and they held on to secure the victory.
Japan lacked the flair that characterised their 2019 World Cup performances and they appeared to feel the absence of loose forward Michael Leitch, who was banned for a red card offence during last week's loss to Samoa.
The win moved Japan to second place in the Pacific Nations Cup standings behind Fiji, who they meet in the final round of the competition next week at the Prince Chichibu Stadium in Tokyo.