A downside is that Rust programmers spend a fair amount of time, at least when they’re first learning the language, figuring out how to appease the compiler. Program behaviors that might have passed unnoticed in other languages (and possibly caused memory errors at runtime) cause the Rust compiler to stop cold.
The other big downside is that you can’t opt out of this behavior. You can’t toggle off Rust’s borrow-checking behavior the way you could, say, disable a code linter for another language. It makes for better software in the long run. But the immediate cost is a language that’s both slower to learn and slower to iterate in.
Rust does allow you to fence off parts of your code with the unsafe keyword, and lift some restrictions, like the ability to dereference a raw pointer. But unsafe does not turn off borrow checking entirely, or anything like that. It’s for taking code with certain behaviors you don’t want casually used (again, like dereferencing a raw pointer) and “gating” it for safety.