There's reports all over the internet over the years up til now, of people saying defragging their modern macs is beneficial.
There is a lot of stuff in the internet that simply is not true. Most reports like that come from Windows users who are migrating to macOS and bringing habits from that environment to the macOS environment. That's not to say a really badly fragmented drive wouldn't benefit from defragging, but you said your drive is 1) External, 2) 4TB size and 3) only uses 650GB. And those three facts say that it didn't need defragging (it's not very full, so there is a ton of free space on it plus, external drives might need defragging, but the interface is the throttle on drive performance mac more often than fragmentation). In your situation, defragging is totally not needed. As for how long it might take, open Activity Monitor, look at the Disk tab to see if read/write actions are underway. If not, look at CPU tab to see if anything is not responding (red lettering) to indicate it's stalled.
Do you have a backup for the external drive? You might be better served just stopping the defrag operation, erasing the drive and restoring from the backup. If you do NOT have a backup, quitting the defrag is less optimal because you may lose the data. In that case, unless Activity monitor show that it's stalled, I'd let it run and treat it as a lesson learned.