Mortgage finance giants Fannie Mae and Freddie Mac may have suffered more than $3 billion in losses due to manipulation of the benchmark interest rate known as Libor, according to an internal memo by a federal watchdog.

The estimate was provided in a memo obtained by Reuters that was sent to Freddie and Fannie's regulator, the Federal Housing Finance Agency, by its inspector general. The watchdog urged the regulator to consider whether or not the losses warranted a lawsuit against the banks that set Libor.

"We conducted a preliminary analysis of potential Libor-related losses at Fannie and Freddie and shared that with FHFA, recommending that they conduct a thorough review of the issue," a spokeswoman for the inspector general's office said when asked about the memo.

Dozens of U.S. and European banks are under scrutiny for allegedly rigging Libor, which has an impact on borrowing costs throughout the global economy. Libor is intended to measure the rate at which banks lend to one another and is used as a benchmark to set borrowing costs on financial instruments, including derivatives and mortgages.

The Wall Street Journal first reported the finding of the inspector general earlier on Wednesday.