Yes, as applied to the current generation of AIs, "safety" and "alignment" refer to things like preventing the product from making jokes about women or ethnic minorities, but that is because the current generation is not powerful enough to threaten human safety and human survival. The OP in contrast is about what will happen if the labs succeed in their stated goal of creating AIs that are much more powerful.
I think we all know this is going to happen, though.
We have AIs that are capable of self-correcting the code that they write, and people have built automatic interfaces for them to receive errors that they get from compilation.
We also have interfaces that can allow an AI to use a Linux terminal.
It's not a stretch to imagine that somebody out there is at this very moment using these in a way that would allow an AI to be fully autonomous with creating, running, and testing its own software using unit tests it wrote itself. And while the current status of AI means that such a program is likely to just not work, you have to admit that we are very, very close to the threshold point where it will work.
This on its own is not enough to threaten human safety, but toss in some bad human decisions...
