Transcript 0:00 Welcome to the debate. You are listening to We The People In Tech Or Not? the debate. That's right, and, uh, just to set the stage here, this is an AI-generated podcast. 0:09 It's based on a post from the We The People In Tech Or Not? newsletter by Alex Rosa. Yeah, and our goal with this show is really to help listeners learn new ways to explain technical topics, right? 0:19 Taking it from a, a beginner, sorta non-tech understanding, all the way up to an expert level. Exactly. So, uh, what are we getting into today? 0:27 Well, imagine you just spent, let's say, two million dollars on a state-of-the-art cloud database. Okay. Massive investment. Huge. And the vendor hands you the manual and says, "You know, do exactly this." 0:38 But on day one, you realize that following their rules is gonna completely break your company's 10-year-old legacy reporting system. Oh, wow. Right? Yeah, that's a nightmare scenario. Right? Mm-hmm. 0:48 So do you halt development for three months to rewrite all that code, or do you, uh, do you void the warranty and just go rogue? That's the tension we're unpacking today. 0:57 So basically, what should a business or a tech team do when they need to choose a path or a setting that the vendor explicitly does not recommend? 1:05 Yes, exactly, and I'll be defending the newsletter author's core thesis here, which is not recommended isn't forbidden. 1:12 I'm gonna argue that with proper justification and strict documentation, deviating from those vendor defaults is a valid and, honestly, sometimes unavoidable business choice. 1:21 And I'm definitely taking the other side on this one. 1:23 I'm looking at this through the lens of, you know, a busy manager or a non-technical stakeholder, and I am highly skeptical that the risks, the operational costs, and the immense documentation overhead of going off the vendor-approved path are actually worth it. 1:37 Well, I think we can- I just wanna know if this philosophy is practical for a standard business or if it's just a guaranteed framework for creating massive technical debt that's eventually gonna, you know, crush your engineering team. 1:47 Fair enough. Let's, uh, let's break down how this happens in practice. I, I wanna start by framing how vendor recommendations actually work. 1:54 Drawing from the author's experience, vendor defaults, even those from massive tech companies like Microsoft, they're fundamentally based on telemetry. Right. So data they're collecting from users? Exactly. 2:04 They look at real cases and past experiences across thousands, maybe millions of customers. So those defaults represent the safest average path because they describe behavior the vendor has already seen and solved for. 2:17 Okay. Makes sense. But the key word there is average. Every single business has unique constraints. So when a customer asks a vendor, "Hey, is this okay to do?" 2:26 the answer usually falls into one of three distinct categories. Let me guess. The hard requirement, the documented alternative, and, uh, the open scenario? Spot on. Yeah, the first is a hard requirement. 2:40 The, the conversation stops there. If the vendor says a certain configuration will actively destroy your data, I mean, you don't do it. Yeah. Hopefully, that's obvious. Right. 2:49 Second, there's a documented recommendation for a different way. So the vendor might say, "Well, our position is to recommend X, but the decision to do Y is yours." And then they actually provide the steps for Y. Mkay. 3:00 But the third category is the most interesting. That's the open scenario. The vendor does the research and finds no explicit evidence that would make them tell the customer not to do it, even though it isn't the default. 3:11 Hmm. My position is that if the scenario is open, deviating is completely acceptable, provided you create a rigorous one-page decision record. See, I come at this from a completely different angle. 3:22 If we acknowledge right out of the gate that vendors base these defaults on telemetry from millions of deployments, choosing to ignore them seems, frankly, incredibly reckless. Reckless? How so? 3:32 Well, we're talking about ignoring the engineers who literally built the system. 3:36 If a vendor has telemetry from a massive global scale and you only have the, uh, the limited experience of your one single environment, betting against the vendor is a massive risk. 3:47 You're playing the odds with critical infrastructure. I see the logic, that I do, but it's not about betting against a vendor. 3:53 It's about addressing a specific business context that the vendor's generalized telemetry simply cannot account for. But look at the actual real-world cost of what you're proposing. 4:02 Let's use the specific example from the post, right? Setting an Azure SQL database to local time, like Central Standard Time, instead of the default global standard time, which is UTC. Yeah, that's a perfect example. 4:13 So think of UTC as a universal currency that works everywhere in the world. Like gold or something. Yeah, exactly. If your system runs on UTC, you never have to worry about exchange rates, borders, or time zones. 4:25 Local time, on the other hand, is like regional cash. It only makes sense in one specific geographic place. Right, and it's useless if you cross the border. Precisely. 4:34 Now, why would a business choose regional cash over universal currency? The post lays out the business benefit pretty clearly. 4:41 Imagine an older application was moved from a local physical server into the cloud, and its built-in reporting functions explicitly expect local time. Okay, so it's a legacy thing. Right. 4:50 If you force that database to use the universal currency, UTC, you have to completely rewrite the application code so the reports still work. 4:58 So changing the database setting to local time saves the company from having to fund and execute a massive code rewrite. Look, I get it. 5:05 I understand the desire to save money on a rewrite, but you're treating database corruption as a minor inconvenience. I wouldn't call it database corruption. 5:12 You're trading a short-term coding convenience for a catastrophic long-term risk. What happens during daylight saving time? Okay, let's explain the mechanics of that for the audience because it is crucial. 5:22 When you use local time in a region that observes daylight saving time, the clock falls back an hour at 2:00 AM in November. Which means the hour between 1:00 AM and 2:00 AM happens twice. Exactly. 5:32 You suddenly have an hour of duplicate timestamps in your database. Yes. Your database can't tell if a transaction happened before the switch or after the switch. 5:39 You've introduced complete chronological ambiguity into your data. Well, yes, for that one hour. And that exact data corruption is what breaks a backpack export. 5:48 For the non-tech listeners, a backpack export isn't just a simple file copy. It's like taking a complete backup of your database to move it elsewhere, like copying all your phone's data to the cloud. 5:58 It's a complete architectural snapshot of your database structure and its data. That's a great analogy. Thanks. 6:03 And when you have overlapping timestamps, chronological sorting breaks down, and the automated validation for that export completely fails. 6:09 So sure, you've avoided rewriting some legacy report code, but in exchange, you've broken your ability to back up or export your own database. 6:16 To any non-technical manager, that sounds like an appalling, indefensible trade-off. It sounds appalling if you assume the team is just changing the setting and walking away. Which they often do. 6:26 But that's not what the author's advocating. The vendor provided the option for local time zone support for a reason. They know businesses have legacy applications. 6:35 If you're aware of the daylight saving time risks, you manage them. Manage them how? The author's framework specifically includes a mandatory section on how we watch for this exact database example. 6:45 The mitigation strategy is to run rigorous tests on your database exports, specifically during the daylight saving time dates in March and November. 6:53 You don't just hope it works, you prove it works during those friction points. But that requires constant flawless vigilance. 6:59 I mean, testing exports every single time, running special engineering protocols in March and November, that's permanent operational overhead. It is overhead, I'll give you that. 7:08 But compare that operational overhead to the immediate capital expenditure of halting your entire development team for three months to rewrite a legacy application that otherwise works perfectly fine. 7:18 It just feels like a trap. It's a calculated business decision, and this is exactly why the author's mandatory step is non-negotiable. The one-page decision record. 7:27 If you take a calculated risk that goes against the default, you must write it down. Okay, but I'm highly skeptical that a piece of paper in a corporate wiki actually solves this problem. Why not? 7:37 Because in the real world, we have massive turnover. People leave, documents get buried in outdated SharePoint drives, and suddenly a team is stuck with a highly customized fragile system nobody understands. 7:47 You're asking a business leader to accept a structural architectural risk today based on the fragile hope that a new engineer reads a specific document three years from now. But the document isn't just a casual note. 7:57 It's a formal operational contract. The author's incredibly strict about what it must include to survive that exact turnover you're talking about. Right, let's hear the criteria then. 8:06 It has to include the date, the specific business benefits you're getting, like keeping those existing reports functional, and the technical risks. 8:14 But critically, it must include the first and last names of the people involved on both the customer side and the vendor side. Yeah, that specific detail really stood out to me. 8:23 Demanding the first and last names of everyone on the vendor call. It's vital for context reconstruction. 8:28 The author explains that documenting, "I talked to a colleague," or, "Vendor support said it was okay," is completely useless. Oh, absolutely useless. Right? 8:36 When a system breaks two years later and a manager asks, "Who approved this?" If the answer is, "I don't know," you hit a dead end. 8:42 But if an engineer on your team got advice from a specific vendor representative, you need that representative's actual name. Even if that person left the vendor? 8:50 Yes, because there might have been a highly technical, perfectly valid reason for the deviation that is just lost to time. 8:56 Having those names means future teams can actually follow up and reconstruct the logic behind the architecture. Look, I see how that creates accountability, but let me put my business leader hat back on for a second. 9:07 Does a manager really need to police a team wiki page for every single toggle and switch a tech team flips? Well, no, not every single one. 9:15 Because if my team has to write a formal deposition every time they configure a server, our development cycle will grind to a halt. It feels like bureaucratic paperwork masquerading as engineering discipline. 9:25 You're assuming this applies to every setting, which it doesn't. The author explicitly notes there are many settings in any given system that you can change from A to B to C, and it has zero systemic impact. Like what? 9:37 Like we aren't demanding a decision record for changing the color of the user interface, right? 9:42 We apply this exclusively to settings that deviate from explicitly documented vendor recommendations, the structural settings that cause real trouble. Okay, fair. Here's a way to think about it. 9:53 Technical debt isn't just bad code, it's undocumented decisions. A non-standard setting with a rigorous decision record is an asset. You know why it's there and how to manage it. 10:03 That exact same setting without a record, that is technical debt. Okay, assuming a team adopts this rigorous documentation today, that's great for new projects. 10:13 But what about the massive legacy systems we inherited yesterday? 10:17 How do you expect a non-technical manager to police a system that's already full of undocumented changes made by people who left the company three years ago? 10:24 That's where the author's second habit comes in, which is auditing non-default settings. You don't just write new documents, you actively investigate the past. So digital archeology. Basically, yeah. 10:35 The post highlights a fascinating diagnostic tool for this. The author provides a specific SQL query that targets a system view called sys.database_scoped_configurations. 10:45 It filters specifically for values that are not the default. Okay, for the non-technical listeners, what exactly is a database scoped configuration? 10:53 It's essentially the control panel that dictates how the database executes queries at a granular level. Think of it like altering the fuel to air mixture in a high-performance engine. Right. 11:02 So a mechanic tweaking it for a race. Exactly. If a mechanic tweaks that mixture, the car might accelerate faster on a specific racetrack, but it might completely stall out when you're just driving on the highway. 11:13 This SQL query basically scans your database engine and returns a list of every single internal setting that has been tweaked away from the vendor's factory baseline. 11:21 And the author states that for every single row that query returns, you need to ask the team, "Why was this changed?" Exactly. And the responses you get are often terrifying. 11:31 You hear things like, "Honestly, I don't know," or, "It was like that when we got here." Or my personal favorite, "Steve changed it," but Steve hasn't worked here since twenty twenty-two. Yeah. The classic Steve defense. 11:43 Right. [chuckles] The author points out that each of those answers is a massive red flag, especially for settings that alter core behaviors. 11:50 If you ask a team why a foundational setting is changed, and they don't know, you are essentially flying blind. You have lost control of your infrastructure. 11:59 I mean, I won't argue that I don't know is a terrible answer in any department. Operating blindly is objectively bad business. But I really want to push back hard on the cost benefit analysis of this investigation. 12:11 How so? If an application is running smoothly right now, spending expensive engineering hours chasing down the history of legacy settings is a massive waste of resources. 12:21 Wait, how is understanding your own infrastructure a waste of resources? Because of the economic reality of engineering time. 12:29 Let's say we run that SQL query on a mature business database, and it returns fifty different non-default settings. 12:34 The system works fine today, but finding out who changed those fifty settings and why, that could take a senior engineer three weeks of digital detective work. Yes, it might take time. 12:44 Why would a manager burn thousands of dollars of budget chasing ghosts in a working system when that engineer should be building new products that actually generate revenue? 12:52 You're calling them ghosts because you're assuming the system will remain static. But in modern cloud technology, environments change beneath you constantly. Sure, but- The vendor pushes a mandatory security update. 13:04 The platform scales up to handle more traffic. You're treating this database like it's a finished building, but it's not. It's functioning infrastructure, though. It's functioning today. 13:13 But let's use a structural metaphor here. You're treating this database like a load-bearing wall in a house. 13:19 If a previous owner removed an internal support beam and didn't document it, the house might look perfectly fine right now. Right. 13:25 But the second you try to add a second story, like a major software update or a cloud migration, the entire structure collapses. Those fifty mystery settings are fifty potential failure points. 13:35 But you're essentially demanding we rip open the drywall to check the beams just in case we ever wanna build a second story. Yes, exactly. 13:44 If you are running mission-critical operations on that infrastructure, you have to know if the beams are intact. Look, the author is very clear. You don't have to panic over every minor tweak. 13:53 But for settings that alter core logic, you must investigate. It just sounds so expensive. 13:59 The cost of a senior engineer spending two weeks mapping out those mysteries is microscopic compared to the cost of a three-day total system outage because an undocumented legacy setting conflicted with a routine cloud update. 14:11 Okay. I see the business case for avoiding a catastrophic outage, and I think this brings us to a very fair takeaway for the audience. 14:19 We can definitely agree that operating a system filled with unknown, undocumented customizations is a major unacceptable business risk. Absolutely. 14:27 Like, "It was like that when we got here," is not an engineering strategy. It's just a deferred disaster. You simply cannot build a secure, scalable future on a foundation of mysteries. 14:37 I'm glad we found common ground there. The, the necessity of transparency isn't really up for debate. But where the nuance really lies and what listeners need to take away is the execution. 14:47 Whether you should actually deviate from a vendor recommendation depends entirely on your team's operational bandwidth to monitor those specific risks over time. Yeah, it's highly situation dependent. Precisely. 14:58 Returning to that local time example, if you change your database to local time to save money on a code rewrite, you must possess the operational maturity to test those automated database exports, specifically during the daylight saving time shifts in March and November. 15:11 Right. You can't just set it and forget it. You have to document it, test it, and review it when new features reach general availability. 15:18 If your team does not have the bandwidth, the budget, or the discipline to monitor that risk year after year, then you have no business taking the risk in the first place. 15:26 You should just stick to the vendor's default recommendation. The freedom to customize requires the discipline to document. If you lack the discipline, you forfeit the freedom. 15:35 That is a highly pragmatic way for a busy manager to view this. It shifts the conversation from, "Can we do this?" to, "Can we afford to maintain this?" Exactly. 15:44 It prevents technical debt from quietly accumulating in the shadows while everyone is, you know, looking the other way. Which leads us perfectly into a very practical next step for our listeners. 15:54 The author gives a Friday homework assignment that applies far beyond just databases and SQL queries. Okay, what is it? 16:00 Any time you go against the advice of someone who has seen more cases than you, whether that's a software vendor, an external consultant, or even an internal senior architect, you need to keep a record of why. 16:11 The homework is simple. Pick just one setting in your system this week that isn't the default, just one, and write three lines about it. Who changed it, when, and why. 16:20 And if a manager asks their team to do this, and nobody can write those three lines, if nobody has the answer? 16:26 Then you just found the agenda for your next team meeting because you have a mystery on your hands, and mysteries in critical infrastructure are ticking clocks. Yeah, that's a scary thought. It is. 16:35 At the top of the show, we talked about how a vendor manual provides this comforting illusion of precision. We want enterprise technology to be binary, right? 16:43 Broken or not broken, strictly recommended or completely forbidden. But it never is. No. In complex real-world systems, that manual's just a guide based on the average telemetry. 16:54 The real operational environment is messy. It's entirely up to you and your team to bring the precision back into your system through rigorous documentation, intentional decision-making, and continuous auditing. 17:05 You simply cannot afford to run your business on I don't know. A non-standard setting with a decision record is an asset. Absolutely. Without one, it's just a liability waiting to happen. Thank you for joining us today. 17:15 We encourage all our listeners to explore these concepts deeper and apply them to their own systems. You can read the full post for free at intecornot.com. That is intecornot.com. Have a great week
