Money Talks: A Tale of Two Change Programs

Like many in the Agile community — or what sometimes gets called the Agile industrial complex — I am all for piecemeal adoption, small-scale before large scale, getting good at doing stuff then expand it out… — what else would you expect from Mr. Diseconomies of Scale?

The “start small” and grow might even be regarded as the canonical approach to Agile implementation. But from time to time I run across something that makes me wonder…

Four or five years back I got involved with an Agile transformation program at a large financial institution, not a bank, more of a mass market asset manager. I was attached to a small team trying to make the whole company Agile.

The coaches viewed themselves as a guerrilla movement, changing the organization from within. They had some success; there was a bunch of stuff the Agile teams and the coaches were doing wrong, but that is another story. This was a licensed insurrection.

As is often the case, this team found it lacked the ability to ask the big questions and get people outside the team (often the higher ups) to engage or change themselves. The organization wanted Agile down in the engine room — at the code face — but they didn’t want to rethink how they set requirements and approached projects. The whole organization was chronically project-driven, obsessed with long-term planning and offshore development. Economies of scale thinking ran riot.

Because the Agile change was at the team level, the Product Owners lacked authority to make real decisions — like not delivering functionality. Yes, the organization wanted to “be Agile,” but the management cadre didn’t see any need to change their own behavior.

One day I met two men who ran the company’s “Software Process Group.” They were guardians of the formal process and “working practices.” My immediate reaction was that they wanted to kill Agile and stick with ISO-9000, PRINCE2, CMMI, and certified approaches. They scared me. But actually, they were very clued up. They got Agile. They saw it was better than the current process.

These two had no role in the Agile transformation; their role was to ensure the company kept its CMMI level-two certification. This was really important to the company because this allowed the company to do business. They told me a story…

During the previous 20 years, the company had grown large, very large, by buying up competitors and companies in related markets. These companies had been thrown together and costs stripped out. One day the financial regulator came to the company and said:

“We have been examining your IT functions. They are not fit for purpose. If you don’t fix it within 12 months we are going to withdraw your license to do business.”

Shit hitting the fan doesn’t come much bigger than this.

The company went to IBM and said, “Help! Fix it — we’ll pay anything.”

IBM flooded the company with people. IBM imposed a process — a traditional CMMI-compliant process. IBM changed the company, not just the programmers but everything. The company did as IBM told them.

And don’t imagine it was cheap. I bet that the change and IBM fees were on the agenda at every board meeting during that period. The men I had met were the remnants of that programm, they worked for the company, not IBM, and their job was to ensure the company maintained accreditation and the financial regulator wouldn’t have cause to come back.

Now contrast this with the piecemeal, small-scale, bottom-up change that us Agile folk are so fond of. Time and time again we get stuck: “The business won’t change,” “We can’t get access to the senior people,” “Existing processes and expectations are unassailable,” “Projects are killing us,” and so on.

I’m sure IBM faced many of those same problems but they had one big advantage: they were expensive.

OK, they had a second: the loss of license would destroy the client company. But when threatened, people often respond by sticking with what they know, so maybe it was a double-edged sword.

Because IBM was expensive, they had access to anyone they wanted access to. Because they were expensive they had authority. And if someone didn’t want to make the changes IBM, suggested then IBM could simply ask the next person up.

Once again, money is information: by spending lots of money with IBM, the company was signaling it really wanted these changes to happen.

Agile changers may not like big change; they may point to the inherent risks, they may point that use of authority conflicts with self-organization, they may understand that diseconomies of scale rule and they may point to a bunch of other risks.

But they should also note one clear advantage: a big expensive change programm brings authority all of its own.

Agile in Private Equity

It’s been almost a year since I joined Mekong Capital. Six months as a advisor and now six months more as a CTO-in-Residence, which in Mekong Capital we call it Value Optimization Board (VOB) member. I am  the technology guy: I love Java, startups, coding, building product, and building great teams. This decision of me joining a private equity (PE) dazzled many of my pals; some think that I am going to retire, others tghink I am downgrading myself from high-end technologies to basic old-fashioned systems. Allow me to explain the context as to why a PE firm and an Agile software engineer like myself might seem an oxymoron.

PE firms are typically known for robust oversight of their portfolio companies, emphasizing metrics-driven decision-making, high margins, and EBITDA. PE firms rely on robust planning processes, sometimes even at the expense of growth, because of the significant leverage and needed predictability. Management teams enter every year with firm plans, the results of which will determine not only their compensation but also whether a bank will be breathing down the company’s neck or not.

Traditionally, waterfall development processes have served the PE community well, as a lot of time is spent in annual planning processes. It isn’t uncommon for Agile software engineers to spend months planning, designing, and estimating future product releases that are quarters away, even though many of those features will never be delivered and requirements will certainly change during the intervening months. This common, fallible practice creates waste in the ecosystem. Agile development strives to delay major investments until they are necessary and eliminates waste in doing so.

So, my existence in a PE firm is definitely not similar to any of my personifications. The challenge is not only the PE firm itself but the investees, too.

  1. Typically the PE investees are laggards in term of technology, Lean, design thinking, and Agile in general.
  2. Sometimes, they feel proud as they just have ERP and some reports (which they like to call BI).
  3. They call themselves data-driven when what they are actually report-driven. Meaning they get a report and make some decisions; there is no experimentation, pivot and preservation.

The way we engineers used to work in product or startup companies very much follows design thinking, Lean startup and Agile. The following figure shows it very well, how these things work together well. I implemented these methodologies with amazing colleagues of mine in two companies in Vietnam called VietnamWorks and iCare Benefits.

Staff Augmentation is as Staff Augmentation Does

I’m in the process of drafting a post entitled “What Do You Know That People Would Pay You For?” But what I’ll put here in this post, combined with the material for that one, is shaping up to be long. So I think I can carve off an initial, coherent point here about staff augmentation.

That one figures to be uplifting. This one? Perhaps not so much. But I think it’s important to establish a premise.

If You Write Code in Exchange for A Salary, You’re Either Staff or Staff Augmentation, Depending on Who Signs Your Paychecks.

Now, for those of you that have worked for product/service companies with a software component, you’re probably shrugging and thinking “Yeah, no kidding, I’m staff.” Ditto those of you who have toiled in a cost-center capacity, maintaining some internal software the company would sooner eliminate.

But those of you that work for custom app dev agencies are probably feeling a little huffy, since most places that sell custom app definitely (i.e. staff augmentation) go out of their way to state righteously that they most certainly DO NOT do staff augmentation. Bear with me, though, all of you.

I don’t actually think there’s anything wrong with staff augmentation. In fact, I think it’s a substantially better model, in most cases, than staff. In accordance with the spirit of Developer Hegemony, I think we, as an industry, should strive to move from staff to staff augmentation, at least as an initial step.

Kafka Streams: Is it the Right Stream Processing Engine for You?

In an earlier blog post, Democratizing Analytics within Kafka With 3 Powerful New Access Patterns in HDP and HDF, we discussed different access patterns that provide application developers and BI analysts with powerful new tools to implement diverse use cases where Kafka is a key component of their application architectures. In this post, we will discuss in detail the streaming access pattern and the addition of Kafka Streams support in HDF 3.3 and the upcoming HDP 3.1 release.

Before the addition of Kafka Streams support, HDP and HDF supported two stream processing engines: Spark Structured Streaming and Streaming Analytics Manager (SAM) with Storm. So naturally, this begets the following question: why add a third stream processing engine to the platform?

With the choice of using Spark structured streaming or SAM with Storm support, customers had the choice to pick the right stream processing engine based on their non-functional requirements and use cases. However, neither of these engines addressed the following types of requirements that we saw from our customers:

  • Lightweight library to build eventing-based microservices with Kafka as the messaging/event backbone.
  • The application runtime shouldn’t require a cluster.
  • Cater to application developers who want to programmatically build streaming applications with simple APIs for less complex use cases.
  • Requirements around exactly-once semantics where the data pipelines only consist of Kafka.

Kafka Streams addresses each of these requirements. With the addition of Kafka Streams, customers now have more options to pick the right stream processing engine for their requirements and use cases. The below table provides some general guidelines/comparisons.

How to Deal With Developers Who Appear to Be Relaxing

The kind of work the developer does is like that of a craftsman, but a usual management mistake is trying to manage the developers like they are in a factory assembling televisions for 8 hours. In those positions, it makes sense to not allow a worker to stop doing their job because then he stops finishing the product, which translates to fewer pieces produced by day, which raises cost and lowers profit.

But that way of management doesn’t work with software developers. The software developers are usually spending 90% of their time figuring out how to solve a problem and 10% writing code. Often, managers with no coding experience think they are doing nothing during that 90% of the time.

I used to be in charge of a developer at a client’s offices on-site and the client complained a lot because, in his own words, “the developer spent most of the day doing nothing.” But when I have on-site developers I always review the daily productivity and he was doing a great job, but the client wanted to always see him writing code during that 8 hours.

In software development, our main tool is our minds. Sometimes it is clouded and makes difficult to do any job, sometimes it’s sharp and we do the work of the entire week in one day, sometimes it’s worried with domestic problems, and at other times it can really focus on the task. Realistically, a software developer is not productive beyond 6 working hours. They needs a lot of coffee (usually), and some free time to get distracted doing something else so we can be productive. Sometimes we are very stuck on what we try to accomplish and we need to talk to a rubber duck to solve the problem.

So, being a software developer is quite complicated. The problem is that the developers’ skills are unique amongst them. Some developers are more proficient in some languages, others on some database engines, others on front end, others in the backend, others on the development tools. So this complicates a lot, trying to unify the way to measure their work.

The work and problems that the developers need to solve also don’t help, as every software requirement is unique. Very rarely and only after many years of experience do we need to solve the same problem twice. So there is not a quick way or shortcuts to developing software. We can take advantage of libraries and frameworks to better deal with the technical part, but implementing business rules on software development is the hardest thing to do. Enterprise software is the hardest software to write.

But this doesn’t mean that you cannot do anything about it. It means that you need to understand the developers, walk in their shoes, so you can create more appropriate ways to work with and a good strategy.

The way that I usually deal with this is by assigning tasks to a software developer that don’t last more than half a workday (4 hours) or more than 2 days (16 hours). If a task seems to be taking more than 2 days, I will break it in two or three pieces or as many as is needed so it will not have a large task where time easily can be wasted.

For example, if a developer needs to do a CRUD, usually the simple ones take about 1 day (8 hours), so I will assign to him 1 task for 8 hours. But let’s say the create part has very complicated business rules and the update part has a different set of business rules which it seems will take 5 days to complete the job. Instead of assigning him one task with the CRUD requirement, I will break it into pieces, like1 task (8 hours) for the data grid, 1 task (16 hours) for the insert, and 1 task (16 hours) for the update. After breaking the main task into 3 tasks, you assign to him 1 task at a time and don’t let him to work on the other two tasks until he finish the first one and so on.

So if by 1 day he doesn’t finish at least 1 task, you will know immediately that something is wrong, he is wasting time or is been blocked by something and is not asking for help. I hope you get the idea. At the end of the day you need to do your homework as a manager, review what is been done and completed on a daily basis.

There is no such thing in software development as x% completed. It’s very common that somebody will tell you, “I spend 4 days in a task and it’s 90% completed, and then I spend other 4 days completing the last 10%.” It is very misleading and arbitrary trying to give task status by percents, as they will be always wrong. The only true way to measure a task is if they are completed or not.

A task will not be finished until a tester or the client (UAT) says the application works as expected, not when the developer says it is done. When the developer says it is done we just put the task timer on pause, and if the tester or the client says it is not right, the time for that task resumes running. On the other hand, if the software does what is expected when we stop the timer and then the task is complete.

How to Use the Spring Config Server

This time, we will use the package Spring Cloud Config to have remote configurations for our applications.

The idea is that our program or programs can move your settings to an external place so that our application is easily configurable and can even change their settings.

This is widely used in microservices. A same service or application can be launched many times in different containers, and it is interesting to have a central place where they can read the settings to these services.

To do this, we will create a configuration server and a client that reads your configuration of that server. The configurations server uses a GIT repository on GitHub, where we store the configuration files.