Building Support Structures

From Support Chaos to Structure: The FundApps Story

Hannes Lowette: Hello, I'm Hannes Lowette and I'm here in London today, and I'm talking to Flavia Circiumaru about setting up support structures in their organization. But I'm first going to let Flavia Circiumaru introduce herself to you.

Flavia Circiumaru: Hi. I've been a software engineer for five years. I've been working on that since I graduated. And I initially came from Romania all the way here. I had a very nice welcome. I'm very excited to be here.

Hannes Lowette: Maybe let's do first things first. You said you were at an organization called FundApps. Can you maybe tell me what FundApps is and what kind of products they build?

Flavia Circiumaru: Yeah, of course. FundApps has been a software company in the regulatory compliance industry for about 14 years. So we have been working on solutions that are basically helping investment managers and financial institutions actually monitor their investments and helping them report their holdings to regulatory entities. That means basically that if they need to disclose anything to particular jurisdictions, we are there to help them manage that and say exactly what needs to be included.

Hannes Lowette: This is the type of disclosure with antitrust litigation where you have a stake in a certain company. Is it that kind of thing?

Flavia Circiumaru: Yes. There are two different types of stakes, and you sometimes also have voting power over those shares. You can make certain decisions over those shares, but more importantly, it helps you understand where you sit in the market, but also it helps the regulators understand where you sit. This is to help, especially with a problem that happened a while ago about 2008, where the financial crisis happened. A big contributing factor was the lack of transparency in the market. So we're hoping that our solutions could help mitigate that and hopefully avoid having our clients actually get fines for not sending those disclosures.

Hannes Lowette: So your clients are mostly investment companies that need to get a view on their portfolio and to figure out where they need to take action.

Flavia Circiumaru: Yes. That's right. We have a few other products, but our main product is called shareholding disclosure. And that particular service is helping us process particular positions of our clients, particularly investment managers that work within financial institutions like hedge funds, pension funds. And they use our system to navigate, especially their positions and assess whether they have to take some sort of action. So the results that they're getting could be as simple a solution as just reviewing those results or generating a disclosure, reporting it to the regulators.

Hannes Lowette: Right. So you have a software solution that helps your customers with that. But you are not building that product. You are in the team that is there between the customer support and the team that is developing the solution. Right?

Flavia Circiumaru: Well, I have been part of the team that is building the solution for the past four years or so. I've then switched to a more leading position where I'm helping the engineering team with support to help fix particular issues that are happening. In the organization, the structure or the process of support is in general, the clients would write some requests and our client success team receives those tickets and triaging whether they need technical supervision or not. But obviously some of these tickets are coming into our territory. And that's when we assess we should actually take action? Is there something that is misbehaving in the service or do we need to have a conversation, perhaps with our product team to assess whether they need to take some sort of action, whether they need to create a pitch for a new feature improvement, or just create some educational content.

Hannes Lowette: Right. So it's a lot of different responsibilities. That's the situation you ended in right now. But you've also been part of the journey towards the current situation, if I'm correct. Right?

Flavia Circiumaru: Yes. The support structure has developed quite a bit since I joined. So I was about six months in when we started considering what our support structure would look like. And back then, we were between 10 to 20 engineers. We grew by about 50% in only one year. And our structure back then was every single engineer that felt that they had confidence in triaging and debugging, they would be part of the traditional support rotation, which meant that you'd be on support for about once a quarter at the time. So you got exposure to the product, which felt nice, but you also got opportunities to see how you can grow the product in different ways, in different shapes that changed over the course of multiple years, where we got more successful in selling our main product, but we also got appetite for more risk appetite for building new features, more services that may not be completely related to the main product. So we felt it was unfair that everyone should be still on support for the main product.

Hannes Lowette: Is that the prime reason why you started putting in place a dedicated team to do this support engineering?

Flavia Circiumaru: It was one of the reasons, but we felt as we were growing bigger and bigger as a team, the context switch was far more detrimental to the overall flow and productivity of these teams. Since we were in a position where we were having micro teams trying a few experiments with Shape Up, we felt it was appropriate to also see how we can change the model of support. So for example, when we were doing Shape Up, and we would work in six week teams, we thought that our support team could also be in a six-week iteration. And that was when initially we thought that we could potentially have a team around it, but since it's only six weeks, they won't feel the dreading feeling that it will be there forever. And they can move on to building more fun stuff. But it didn't really work. I think at that point in time we were not ready to have such small iterations of people being in the support team, because there was a lot in the backlog, and there were still a lot of practices that weren't quite in place to really sustain that amount of work.

Hannes Lowette: So that was a rotation of product engineers, then being on the support team for six weeks, and then rotating. And that's basically replicating the model of having a support rotation. But then for longer periods.

Flavia Circiumaru: Yes. And with a bigger group of people who rotate around.

Hannes Lowette: What were the things that were good about that  ,and what was bad about that? Why did you move on from this?

Flavia Circiumaru: A few benefits were very clear from the beginning. It felt right because it matched the model of all of the other teams that we had currently going on. So it felt like we were in sync with everyone else. Also, it gave some people the opportunity to become experts in particular parts of our main service. But over time, we felt clear that some things were missing, some nonfunctional things. We didn't have the right monitoring in place. We didn't have the right processes in place to continue this cycle on and on. And in general, our engineers that were in support during those iterations felt burned out. It could have been because we weren't necessarily looking into support goals at the time. So the premise was you get a ticket, you react to it, you triage it, and you go on and you do that forever until you're done. And it's not necessarily a nice place to be. But we fixed that, I think in the last year or so.

Recommended talk: Privacy-First Research with OpenSAFELY • Eli Holderness & Hannes Lowette • GOTO 2025

Three Desks, One Goal: Better Support Engineering

Hannes Lowette: How did you fix that?

Flavia Circiumaru: So we decided to put an entire team on support for an indefinite amount of time, which meant no one is changing positions for at least six months. And the structure was slightly different, which meant that we are about four people in the team right now, and one person would stay on the frontline. We call it the front desk, where you respond to every query as it comes. Try to reduce the response time. Triage data requests that are coming in. There will be a person on the second desk where they have the space to analyze tickets that are more tricky. That may have carried over from previous weeks. Then there's the third desk where you can have one or two people. And that's where the magic happens, I would like to say, because you had the time in that time frame to improve the support burden in some way, you had the time to work on your monitoring, on the observability. So, see what exactly is not working in your support workflow at the moment. So instead of using that space to create new features or just accept small feature requests that are coming your way, you are mostly using that time to fill the gaps in your monitoring and your observability, and your processes. And that's paid off over time.

Hannes Lowette: So it's a fixed position that you freed up to patch any holes in how you operate your products.

Flavia Circiumaru: Yes. I think the fact that we're still rotating in those positions within the team really helps freshen up the minds of everyone around. And I think that's what keeps everyone going. They don't feel burned out. They feel like they contribute in some way.

Hannes Lowette: What sort of things have come out of that, having that role in place? Can you have some very specific examples of things that were implemented by people that were working that role?

Flavia Circiumaru: Certainly. We are talking in the context of our main product here, and there are a lot of microservice pieces at play that are quite integrated with the main product. So by extension, we were also responsible for them. And what became clear is that we didn't have any sort of visibility over the health of the deployment in those areas. Whenever a change would happen, they could have gotten stuck multiple versions behind, and it became clear once some people started complaining about it and actually making some fixes around those problems, around those areas that we can put more monitoring in place. A different problem that we are currently fixing right now is the creation of our tenants, the creation of our environments for our clients is still as much as we automated a few elements of that particular category of work, there's still a lot of automation that can be put in place. And there's been a lot of delays because people are modernizing our deployment pipelines architecture, and they felt more appropriate to start with fresh tech stack. So we are now working towards actually building those automations. And I'm very excited about that because it could save us tens of hours every month.

Hannes Lowette: Right. Are the people in the team at the moment still people who rotate in and out of the dev team, or do you just rotate inside the support engineering team?

Flavia Circiumaru: We rotate mostly inside the support engineering team, but since we are in a better position than we were last year, we feel far more confident in just welcoming anyone into the team that want to try our processes. It's also a very nice place to onboard new people that are coming into the company because you get exposed to problems and many different parts of the product also.

Hannes Lowette: How close is the collaboration between that team and the development team? Because I mentioned that you're developing parts that improve the operations. I guess those get contributed back to the code base as well. They push new versions your way, which you're then supporting. I've seen in my days a fair few situations where that causes friction. How do you deal with that inside your organization?

Flavia Circiumaru: Yes. In our case, we collaborate daily with our dev team. Especially we have regular checkups with all the teams that are sharing the same code base as us. I think that's how we navigated the fact that we do share our code base. So we are aware of everything that's happening most of the times. There is also an aspect of the engineering culture that we have. We are very vocal about significant changes that are happening across, and we had our share of painful events where we had to understand how we can avoid incompatibilities between versions and actually understand if we need to rebase, and a lot of aspects that could block us from each other. So I would say the thing that really helped us was over communicating a lot of the time, over collaborating in particular, as we are in a position where I like to call ourselves the conductor of the orchestra in the sense that we're supervising a lot of the other underlying processes and projects that are happening. I like to be in regular contact with some of the team leads to understand what the current process is and what we should be expecting to happen next.

Hannes Lowette: Because if I want to summarize this, in your team, in your role, there's a whole bunch of different things passing through it. You have customers who have contacted you with issues, and those can make it to your team. But also things like feature requests pass through your team. Sometimes you would then decide, hey, this is a misunderstanding, and you communicate it to the team that is making the training content for your customers. And you're a bit like the spider in the web pulling all the strings, right?

Flavia Circiumaru: I do like to think of it that way.

Hannes Lowette: So but that also makes it interesting because it means that you also can impact a lot of the parts of the business, not just the dev team, but how have some of those things played out and how have you changed the processes when you're working, for instance, with the team that makes the training content?

Flavia Circiumaru: Certainly. One example comes to mind when you are asking me this, which is precisely a part of the product that was a functionality of the product that was used quite differently to what you would expect by the clients. So there was a particular page on our platform that was aggregating all the potential data that the client has put into our system, and our client thought, this is the best place to take all that data back and aggregate it and use it in my company, because there are different aggregations within my company. So I can have an eagle eye view and it was particularly a very low-performing endpoint on our site, since we were not expecting that page to be used that way. Initially, we thought, oh, we're going to use pagination or caching. We're going to find different ways of fixing this. But after several conversation sessions with our product team, we actually realized that it's far better to communicate with our clients and understand what they really need from that page, because that page was not meant to be used that way. And it was a very lovely moment when we understood that we actually don't need to fix everything that our clients are saying, but actually have a conversation with our product team. And in that moment, we realized that we have far more impact than just fixing things that are coming our way.

Another aspect of this job that I like is related to the documentation piece that you are alluding to. We realized that a part of the job is not just helping our clients implement our solutions, but understanding how our client services understand our documentation and how easy it is for them to do their job, to do our job by themselves. Because we currently have a lot of documentation in place, but some of it may not be as clear as one would think. And if we were just adding a few bits and pieces and spending a bit more time creating workshops that could help them understand how they can implement with a particular client, it could help us also reduce the support burden that we have. We would have far fewer support requests coming our way of that sort.

Recommended talk: Organizational Sustainability with Platform Engineering • Lesley Cordero • GOTO 2024

The Challenge of Multi-Tenant Support at Scale

Hannes Lowette: Right. From not having a dedicated support engineering department like everybody picking up the role for a week or a few weeks and then rotating through the team, until you got back at the start again, to now having a dedicated team that is the spider in the web and trying to connect basically the customer facing part of the business. Am I putting that correct?

Flavia Circiumaru: Yeah. That's right.

Hannes Lowette: So we've gone from zero to hero. Over quite a bit of time. What are the hurdles that you really had to get over? Was there a moment where you thought like, this isn't working? We're doing it all wrong. We have to do things differently. I'm looking to see what I can learn from your journey in that regard.

Flavia Circiumaru: Yeah. There were definitely a few bumps in the road. I would say initially, in the beginning of our journey, it was particularly difficult to understand what the business needs from such a team because we felt the need to be far more autonomous than we initially expected. When you don't have a support team, when you never had the concept of an internal team that's facilitating this, pulling strings around, it's quite difficult for the leadership to actually understand what our purpose is? So they could; it was very easy for them to interpret us as a team that could just build other things. So they would throw a lot of feature requests at us, expecting us to do all of the things that the product managers didn't have time to put on their roadmap.

Hannes Lowette: Another dev team.

Flavia Circiumaru: Yes. How did you push back against that?

Hannes Lowette: It was a lot of conversations that had to happen to understand, to make clear what the current burdens are. The problems weren't that we didn't have enough people to build new features. The problem was that there wasn't anyone to maintain the existing features in a mature way with the right standards in place. So we had to assess exactly where we stand with our current services. There wasn't a lot of information about how to track past support tickets. What exactly were the pain points? We didn't even know. What were the pain points? We just knew that our issues are piling on and on, and that felt enough to get a bit more freedom into what we decide to do.

I heard you mentioned two things that seemingly contradict each other. On the one side, you just mentioned freedom for the team. On the other hand, you mentioned that you were evaluated on what the values of the company were to align with that which basically suggests less freedom because we're going to align ourselves in the direction. Is that something that has been a struggle or did that magically just line up, or how did that work?

Flavia Circiumaru: I think I don't necessarily see values of a company coming in contradiction with freedom. It's more about aligning the team's expectations of what they should be looking for, what they should build towards. Because there are many different ways in which you can transpose basically initiatives into actual business objectives. So back when we started, one of our goals was getting into a profitable state. So understanding how we are consuming our resources, how we can reduce that amount of time, how we can basically see ways of improving our performance and becoming more efficient. But on the other side, our main objective was keeping the lights on for our customers, making sure that the response time is nice and low, having a great relationship with our customers and understanding that they are listened to and that the pulse is there constantly. And that helped enough because it felt like the values are at a high level enough that a lot of elements could be put into place to build towards gaining that goal for the company.

Hannes Lowette: So, keeping your customer in mind throughout that whole process is super important as well.

Flavia Circiumaru: Yes.

Hannes Lowette: Do you have any metrics that you measure, because ideally, as a customer, what I would want if I call any type of paid support, it doesn't matter if it's FundApps or anyone else. I will want somebody to quickly pick up my case. Somebody who knows what they're doing, knows what they're talking about, and that basically owns the process from there on. They'll take care of it, that either I get an answer or my problem gets solved. Right. That's the ideal world. Now, that would require infinite resources. So probably there's going to be a trade-off here. What are the values that you measure against when you decide we're going to put more resources into this process? We're going to now prioritize this. How do you deal with all of that?

Flavia Circiumaru: We had a few metrics in place, as you were saying. Most of them were around our response time. A lot of satisfaction was brought just by the fact that we have responded to the client's requests. But the way that we would prioritize certain tickets over the others is essentially what is the impact? The current impact of the problem is something that only this client in particular is facing, or is actually a problem that nobody else has raised. But it's happening across all of our environments. And in general, if there is an opportunity for growth, we would assess whether it is the right time to go for it or not. So for those pieces that we are not quite sure exactly where they sit, we have a lot of tagging in place to have particular categories that we can supervise over time and see how they are growing across the months. And that's our general instinct on how we can pick up patterns. So patterns of issues that are coming our way, that would be one way that we certainly prioritize. But sometimes you have to pick the ones that are for the clients who are shouting the most.

Hannes Lowette: Loud client gets the solution.

Flavia Circiumaru: Yes.

Hannes Lowette: Something interesting to me, as a developer and a technical person, I heard you mentioned bootstrapping your clients' environments, or you just mentioned whether it's a problem that occurs in all the environments. That leads me to assume that every customer has their own deployment of your product. Is that correct?

Flavia Circiumaru: That's right, that's correct.

Hannes Lowette: It gets deployed in their cloud tenants. But it's managed by you. Is it that type of situation?

Flavia Circiumaru: Yes. Every single client has their own account. And that's where we deploy our infrastructure. They get their versions of our applications. But we also have a lot of non-tenanted services that are communicating with us.

Hannes Lowette: The tricky part with that is I'm assuming because this is both very important and mission-critical data to your customers. So it's very confidential. It's also running into a whole bunch of different cloud accounts. Sometimes you get problems that are probably hard to reproduce. How do you deal with that?

Flavia Circiumaru: It's still quite a tricky problem to fix. And we are looking even this year into how we can replicate particular environments without accessing sensitive data. Security is a very big part of that. So we are treating that with utmost respect. But we are currently trying a phase where we are asking our client success team to replicate, to the best of their knowledge, how the system works for the client. So they have access to the client data. Obviously, they cannot disclose it, but there are particular setups so they can replicate, knowing the particular skeleton of their files. So we're doing a bit of trialing. We've restricted environments where our client services can use that kind of skeleton and see how it behaves in those environments and see whether they can replicate the problem or not. But we're not quite there yet. I would like to say there are still a few elements that cannot be truly referenced correctly. But in general, we are trying our best to encourage our client services team to try to replicate the issue first, because you are the most likely to get it to that point rather than us. You have far more access to that data than us. There are certain elements where we still feel like we can access the client environment for the people who have privileged access, who have been background checked, and that's where we can do one-off fixes, but we try to reduce them as much as possible.

Recommended talk: Stop Punching Yourself in the Face • Hannes Lowette • GOTO 2020

Future Aspirations and Advice

Hannes Lowette: I can imagine there's nothing more annoying than having different branches that you deploy to different clients, and then trying to manage all of that in the long run. If you could look ahead, from the situation where you're in right now, what would be one of the things that you would like to work on to improve the processes that you currently have?

Flavia Circiumaru: There are a few elements. One, we just touched on the aspect of our replication. It can be far better. I would like to get into a place where we can use our support team as a tool for educating all our joiners, to understand how our system works.

Hannes Lowette: Joiners in this case being new employees for FundApps?

Flavia Circiumaru: Yes, it can be a very trial-by-fire type of onboarding.

Hannes Lowette: No promise that it was going to be easy.

Flavia Circiumaru: When it should be somewhat challenging, but it should feel satisfactory. They also shouldn't be drowning. They shouldn't feel burned out or overwhelmed at this point. So I would like us to get into that stage, when it comes to other aspects of our engineering practices, I would like to get in a position where our developer experience is in a better place. We are still experiencing a bit of trouble here and there. We have our local setups and our deployments could be even more observable than they are right now. But I would say I'm still quite happy with how far we've come in only one year. And the intention, at a particular stage, whether that is in a year from now or three years from now, is to get into a place where we don't need a support team again. So, having the opportunity to just be on support for a week or two and then try again a new project, that's where we want to get back to.

Hannes Lowette: So basically, you're aiming at going back to the situation that you were in before, basically by making your own job obsolete.

Flavia Circiumaru: Yes.

Hannes Lowette: I'm guessing in that whole process, that role of implementing improvements to the operations, whether it's the deployments or the observability or whatever, that's going to be key in making that happen, if it's even possible at all to get there.

Flavia Circiumaru: Yes. It's a moonshot kind of goal. And we'll see if we can get there. It might be a cyclical project where we get into a place where we can no longer have a team, then decide actually there are a few things that still need to be looked into. But we are currently in a process where we are trying to ingest more and more features under the support umbrella that we have. So that's the currently very interesting part of this work because we are in a position where we can set the standards of great supportability for a service that we never touched. After all, it's very hard for us as people who have a lot of domain knowledge, to suddenly ingest a lot more domain knowledge to understand how to support particular features. So we need to figure out how we can set the standards of great documentation, of great monitoring already created and in place, so that we don't need to do most of the work, we can just use whatever has been handed over.

Hannes Lowette: What kind of tips would you have for people who are going to embark on a similar journey to yours? They're struggling to deal with customer requests. What are the tips that you can give them that they can learn from what you have done?

Flavia Circiumaru: I would say be patient. Initially, there will be a lot of confusion of what you should be looking into first. You should assess as far as possible what type of requests are coming your way and what elements you need to improve first. But the tools or the guides that particularly helped us are setting up our own team goals, understanding what exactly needs to be reached by the end of the year. For us, it was quite simple. Reduce the risk, reduce the response time necessary, having our clients happy and feeling a sense of happiness within the team, that we're heading into the right direction, that for us, felt like it was enough. And as far as anything else goes, you have to be in constant contact with the team that is building your code base to ensure that nothing creeps out of your horizon.

Hannes Lowette: Yes. Who'd have thought communication was such a skill?

Flavia Circiumaru: I know, yes. We have a lot of people who can communicate very clearly, and we have all sorts of communication, but you can never have enough. And there are all types of communications that are happening that you don't even consider sometimes. It's not only the verbal regular catch-ups, it's the documentation that you leave in place, it's the findings that you have, it's the fact that you fix the problem that everyone else was struggling with. And you make it very clear, and you show how you can fix it. Yeah. I'm very proud that we have this kind of culture.

Hannes Lowette: I'm happy to hear somebody talk about the support team in this very warm way, and to hear about a company where this is, instead of being a point of friction between development and the people who pick up the phone, you're trying to pull everybody closer together. And improve the whole company from that point of view. And I think for me personally, if I were to embark on the journey and take something away from what you just told me, it's do that. Be talking to all the other parties involved, whether they be your customers, but also your customer success manager and the support team, and the dev team or whatever it is, you align all of them, good things can happen. And then the friction that is usually there, because I've been in plenty of environments where there is friction between the helpdesk and the dev team because, well, we never get usable tickets or we can never figure out what's going on. It seems like you by putting the team in the middle, you've solved some of that. And that sounds pretty great.

Flavia Circiumaru: Yes, certainly. It's been a very fun project for us. And to see how we can make support a place where everyone feels heard and everyone feels like they contribute to the company, not just react to some tickets that are coming our way. And it's been a very pleasant journey to see how we are also heard back. A lot of the impact that we have made across the year has been our main reason to keep us going. That makes us feel like we're making a difference.

Hannes Lowette: So good to hear. And I wish you all the luck in the world, making your own job obsolete again.

Flavia Circiumaru: Thank you very much.

Hannes Lowette: Thank you for joining me today. And thank you for watching.