You can actually install a trustworthy boot firmware on your computer yourself and actually verify that it really is. The issue is that you would have to do it each time you use a computer this is very unpractical. Since you would need an additional computer to copy it from, it would shift the trust that additional computer, which would have the same issue.
Can you trust your computer if its software has been arbitrarily modified? As the software that runs first, and that is the most privileged, the boot firmware could modify the software that runs after or along it.
Lot of devices using system on a chip can be configured to refuse to unsigned boot softwares. This is however ireversible. The users opting for such scheme would have to either delegate that trust forever or to maintain their own PKI. If they chose the later, loosing the signing key would prevent the installation of new boot firmwares. If the signing key is exposed, it would void the boot firmwares security.
With Chromebooks, the first stage of the boot firmware is typically made read-only. It then checks the signature of the next stage, and enters recovery if it doesn't match. Lot of effort is made to ensure that the users are not prevented to load the operating system that they wish. With some devices, there is even some facility to permit the user to easily run a Coreboot payload such as SeaBIOS. However totally changing the boot firmware requires the user to disassemble the device to remove the flash write protection. Even if this is made easy, it is unpractical for a lot of users.
Another approach would be a (separate) chip that would verify the boot software signature and only release a secret if it matches. The user would be able to change the public key at will. This would also erase the previous secret, and the user would have to re-add one. Such mechanism could be used to display a secret picture to the user, attesting that the system is in a trustworthy state. This picture wouldn't be available if the system is untrustworthy. That way the user would for instance only their passphrase when the picture matches. It could also release an encryption key that would unlock some data partition. The key would be lost if the boot firmware is modified. This way the users would be able to delegate the trust to whoever they wish, and still be able to reverse their decision when they wish.