Metrics are only good when you can disregard some amount of errors without investigation. But they're a financial organization, they have a certain amount of liability. Generalized metrics won't help to understand what happened to that one particular transaction that failed in a cumbersome way and caused some money to disappear.
You can still have logs. What I'm suggesting is that vast amounts of unstructured logs, are worse than useless.
Metics tell you where and when something when wrong. Logs tell you why.
However, a logging framework, which is generally lossy, and has the lowest level of priority in terms of deliverability is not an audit mechanism. especially as nowhere are ACLs or verifiability is mentioned. How do they prove that those logs originates from that machine?
If you're going to have an audit mechanism, some generic logging framework is almost certainly a bad fit.
> You can still have logs. What I'm suggesting is that vast amounts of unstructured logs, are worse than useless
Until you need them, the you'd trade anything to get them. Logs are like backups, you don't need them most of the times, but when you need them, you really need them.
On the flip side, the tendency is to over-log "just in case". A good compromise is to allocate a per-project storage budget for logs with log expiration, and let the ones close to the coal-face figure out how they use their allocation.
Even at very immature organizations, log data within a service is usually structured.
Even in my personal projects if I'm doing anything parallel structured logging is the first helper function I write. I don't think I'm unrepresentative here.
> Even at very immature organizations, log data within a service is usually structured.
unless the framework provides it by default, I've never seen this actually happen in real life. Sure I've seen a lot of custom telegraf configs, status end points and the like, but never actual working structured logging.
When I have seen structure logs, each team did it differently, The "ontology" was different (protip: if you're ever discussing ontology in logging then you might as well scream and run away.)
I suspect you and the parent are using different meanings of the word "structured". They're not totally random or they wouldn't be usable. It's a question of what the structuring principle is.
Am I crazy here? We run all of our app logs and error logs through LogStash and just have a few filters in there to normalize stuff like the timestamp. Honestly the only peace of data that absolutely HAS to be standardized because that’s the piece of data that splits our log indexes, is the primary sorting mechanism, and at what point we roll up an index into some aggregates and then compress and send it to cold storage.
> "But they're a financial organization, they have a certain amount of liability."
In the loosest possible sense. Binance is an organization that pretended it doesn't have any physical location in any jurisdiction. Its founder is currently in jail in the United States.
It's always struck me that these are two wildly different concerns though.
Use metrics & SLOs to help diagnose the health of your systems. Derive those directly from logs/traces, keep a sample of the raw data, and now you can point any alert to the sampled data to help go about understanding a client-facing issue.
But, for auditing of a particular transaction, you don't need full indexing of the events? You need a transactional journal for every account/user, likely with a well-defined schema to describe successful changes and failed attempts. Perhaps these come from the same stream of data as the observability tooling, but I can only imagine it must be a much smaller subset of the 100PB that you can avoid doing full inverse indexes on this, because your search pattern is simply answering "what happened to this transaction?"
The reality is that when their service delays something they owe us tens to hundreds of thousands of dollars. This is the tool they’re using but if they can’t even get a precise notion of when a specific request arrived at their gateway they’re in trouble.
