Mayayana wrote:

In my experience, userAgent sniffing for no good reason has become an
increasing problem, so it's a good idea to spoof that.

It was a quick and very easy way to determine if the client supported
functionality the site needed, or would prefer to use, to present the
content how they wanted. Say you want to use VP9 on your site (see
https://en.wikipedia.org/wiki/VP9). That didn't get support in a
released version of Google Chrome until 25. Maybe something in VP9 was
needed that wasn't supported back in VP8 when support got added back in
Chrome 15. Even if it was, VP support wasn't added until Chrome 15. If
the visitor was using Chrome 14, the UA would quickly tell you that you
either had to adapt to the client (present the content somehow
differently), or tell them then needed a newer or minimally versioned
web client. It would take more code, testing, and debugging to add code
to test the feature set of the client than go by a table of client
versions knowing when a function got supported.

It's easier driving down the street to use the house numbers to find an
address then have to stop by each house, ring their doorbell, and ask
them their address.

I don't see the UA header was a bad idea. It was simply a lookup table.
However, it was when web clients were allowed to lie (spoof) the UA
header that the visitors ****ed up the table lookup, so something else
was needed, and that something else is more complicated. Sometimes you
can query the client as to which features they support, but that's more
code than looking at the UA header already sent to the server.
Sometimes you have to figure out how to write code to test if the web
client will support a feature.

Sure, if web browsers never changed then the UA header would've never
been needed, and we'd be back on Mosaic 0.5 of 27 years ago. That
didn't happen. Because web browsers added more features along with
support with new standards that the UA header became necessary.

Perhaps you forgot how Microsoft attempted to wrest users to using their
Internet Explorer and force web authors to either develop their site for
other clients, only for IE, use the UA header to let the server select
which version of a document to deliver to the client, or create a
****ing mess in the web document's scripting trying to query the web
client as to who it is which is making the web document try to adapt to
multiple different web clients (but not compliant to all of them).
Microsoft wasn't the only one doing it their way making it necessary to
determine which version of a web document to deliver to the web client
or using scripting to adapt a web document to the client. It wasn't
just about a difference in features between clients, but also
differences in how they behaved for the same HTML tag. Without the UA
header, the growing pains would've been far worse and the Web would've
floundered a lot longer while not yet ready for mass consumption. So,
instead of making it easy by enforcing clients to correctly identify
themselves via the UA header, we end up with web documents using
scripting to identify the client after delivery, like using
javascript:alert(navigator.userAgent) and then a code block to select
which part of the web document gets rendered. Of course, those that
disable Javascript in their web client thwart the document-centric
discovery method, and hope the site hasn't fully deprecated the abused
UA header as to completely ignore it.

Back then, web clients did not respond to inquiries regarding their
feature set, so the UA header afforded that identification. It is still
far easier to use the UA header (that all web clients send, anyway) to
use a lookup table to determine if the client will support some or all
functionality employed at web sites. Sites tend to grow into newly
standardized functionality. Users can be slow to update, especially if
still lingering on old versions of an OS. The UA header, something
required (well, strongly recommeneded) by RFC for the client to send to
the server, means the server can use a lookup table to detemine which
version of a web document to deliver to the client.

Too bad web clients were allowed to subvert client identification by
allowing those web clients to lie. If web clients were not allowed to
lie in the UA header, the UA header would've remained an easy way to
identify the client to know what the server should deliver. A good
thing based on a trust model will get abused making necessary other
measures to identify the web client.

If everyone down the block removing their house numbers and all used the
same house number, how would you find a particular person on that block?
With the home owners are allowed to lie about their address, something
else and more complicated is needed for identification. Yep, a trust
model that would've been sufficient if not for the abuse.

Yes, there have been problems when a new version of a web client gets
released whose UA string has not been known before. Same for new web
clients that get introduced that have their own UA header. Takes time
for sites to update UA tables to include changed strings for new
versions or new clients. Almost seems a UA string authority is needed
to which every web client author must register their UA string for every
old and new version of their product. Uh huh, like that would happen.
Web client authors would bitch their release gets slowed because they
have to wait for UA registration, and someone else is involved in the
release of their product. Imagine what would happen with the same
attitude toward domain registration or site certificates. Those would
become worthless. That's what happened with the UA header because there
no authority regulating and recording the UA headers, but just a bunch
of independents trying to provide tables of UA strings for what they
know about at the time.

For a little history, see:

What a ****ing mess from each wanting to do it their own while
impersonating others, and all because there was never an UA authority
regulating the definition of UA strings.