cinch Developer Stories – Greg & Toli

An interview with cinch’s Principle Engineer Greg Farrow and Principle Practice Engineer Toli Apostolidis. 

Hi Greg, Toli! How are you both doing today? Could you quickly introduce yourself and give a quick overview of what you do in your role at cinch and what your responsibilities are?

Toli – Nice to meet you, Alice! Thanks for sending the questions out. Nice to be here with Greg, too – he’s been great to work with 🙂 

I grew up in Greece but have lived most of my adult life in the UK. I have worked in the software industry for 11 years now, with experience across a few industries, and all software stacks. Currently, I am a Principal Practice Engineer at cinch, with a particular focus on Observability and DevOps. 

The ‘Practice’ part refers to Engineering Practices: I help define and evolve engineering practices for all teams at cinch. I focus mostly on practices that optimise our software reliability via Observability, Deployment patterns, Monitoring etc. Ultimately, contributing to what we call the ‘golden path’. This work also contributes to an engineering progression framework we are about to launch. I see my role as a form of Engineering coach, serving the teams to help them work better together. I don’t do this in isolation, though. I lead or participate in Communities of Practice or Communities of Interest. And I initiate or participate in short lived Working Groups that attempt to provide org-wide solutions. Examples of these are ‘DevOps Community of Practice’ or ‘Observability Blueprints Working Group’ (which Greg has led with great success).

It’s worth mentioning that this type of work – creating public content like this – is part of my work. I recently gave a talk at a local meetup on ‘The road to observable systems’ or my talk at dash con last year on ‘cinch’s road to serverless’. All this content was 2 years in the making. It’s weird to say, but when we were only a handful of people in a small office in Manchester, we knew we would be in a position to talk about some great things in 1-2 years time.

Greg – Hello! Thanks for the chance to share some of our experiences at cinch. I joined in August, so I still feel fairly new. The last few months have been an exciting challenge and I have really enjoyed it.

I originally started my software journey in testing, way back in 2008. On that journey I have built a lot of software and tackled lots of different problems, from VOIP to attribution algorithms.

At cinch we have two different types of “Principal” – the “Practise” people are the rockstars, subject matter experts, like Toli. They work across the whole engineering group. The other type of Principal are generalists, like me who focus on a smaller group, 2-3 squads.

My role is really interesting. My focus is on supporting my squads, helping them to work effectively in all things technical. This includes architectural design, code, automation, observability, etc, etc… This can take many forms, from pair-programming to facilitating workshops.

What is the professional progression like at cinch? Toli, I can see that you’ve had 5 different roles at cinch within your time there.

I was the first software engineer hired at cinch. This means that my journey at cinch has followed the organisation’s trajectory. At every turn, I have followed the growth with a new role – usually a role that is relatively new. So for me, there has been a lot of discovery work and path finding which I have really enjoyed doing. The most important thing I have discovered is that due to how our organisation is structured and the way we work, there is a clear path for non-managerial growth. You can progress as an engineer without having to be a manager. I don’t manage anyone in my current role. But I feel like I have plenty of people that I was able to mentor and help grow without the explicit line management responsibilities. 

With cinch being a start up at first and then a scale up, if you join, you are likely to follow the growth of the company and get opportunities for internal promotion that you wouldn’t have at a larger organisation. As I hinted earlier, we are about to launch an engineering progression framework. This will define – in writing – what principles and skills underpin the various roles we have at cinch. This is particularly exciting for me as it makes this unwritten progression visible and explicit. 

Greg, your personal blog mentions you 💛 serverless! Touching upon the ​​serverless architectures, can you talk a little bit more about this and why it excites you?

I’ve alway felt strongly that software engineering is about solving problems for people – not building software. It’s probably because my background originally was less technical, but I’m not a fan of technical elitism and always prefer solutions that prioritise the customer experience, over the implementation “purity”. That’s why I think Serverless is the most exciting technology I have worked with.

Serverless allows us as engineers to focus on the solution rather than the technology. Having used “pure serverless” on a number of products now, I am still amazed at how quickly you can build a quality, scalable, cheap solution.

Combined with some other techniques, such as infrastructure as code, automated workflows (CI/CD) and trunk-based development, it’s now easier than ever to build amazing products quickly and easily.

Toli, do you have any tips on improving observability of the state of running systems – do they use REPLs or just intensive logging?

Oh, I could talk for a very long time about this. Great question. At cinch we believe in teams of people. Cliche, I know. But if we want a particular slice of our software to improve and evolve to a faff-free service for our website visitors, we put a team behind it. And we trust the team to excel. The Principal roles (Practice or not) are there to support the teams in any way they need (i.e. glue work). Same goes for observability. Observability is a property of the system (that we desire our system to have). To make a system observable we need a team behind it that wants (and knows how to) upgrade its observability. Software systems are made observable by teams.

In practice, this means instrumenting the code well. And that means going past logging. We use:

• Tracing extensively for the backend
• Metrics: either custom domain-specific that we emit, or AWS service metrics (or any 3rd party really)
• Logs: custom domain specific that we emit, or logs that we ingest again from 3rd parties
• RUM events: the tracing of the frontend. We link these to the backend tracing

Once the telemetry data is in a sufficiently good place, the usual follow: monitors/alerts, synthetics, dashboards, SLOs etc. etc.

All this is backed by practices and patterns that the teams create and evolve to make sure the observability of their systems is up to scratch at any point in time. Ultimately, they organise their ceremonies and practices so that they can… observe their production software systems. 

A final note is that the core principle of building observable systems is: teams who build software systems know the system best. So they build, they ship, and they operate. If anyone ‘gets’ their systems, it’s them. And we trust them to do it.

The road to observable systems

Toli, how did your background in mathematics and theoretical physics lead you into the career you have today?

Toli – That’s a very interesting question. Until I was 23-24, I had never imagined myself working in software. I was scared of coding. I could understand abstract notions of Algebra or Mathematical Physics. But I never wondered how things *actually* worked. That’s the first mindset shift I had to make when I started my first software job. I got the job in the first place because of my Mathematics background: they were looking to hire Mathematicians or Physicists. ‘We can teach you how to code’. 

I learnt how to code by writing algorithms for large scale optimisation problems: solving the travelling salesman problem, i.e. optimising routes for trucks. It was a fascinating problem domain, and kept me there for 7 years. 

I think not having a software engineering academic background has hindered my progress at times. But I spent years in the world of Mathematics and Physics, learning how to think about concepts, being methodical and systematic. I picked up the cliche skill of ‘attention to detail’. That all helps when it comes to building software. There was one more thing: I was never the nerd Maths guy. I was always sociable. I played basketball semi-professionally. And coached basketball for a few years. So I do understand how to be a team player, how to coach, how to lead when needed, how to listen, how to put my ego under a greater good.

Personally, what are your favourite things about working at cinch?

Toli – I was telling someone recently: cinch was the first workplace where I realised I can be myself. And that’s fine. That may be shocking to some. But I do know – from experience – that workplaces exist where you have to have adopt a ‘work persona’. 

My next favourite thing is that the Leadership who started cinch really did ‘get’ some core engineering concepts. Essentially, the same concepts that make cinch an Exponential Organisation (have you read the book?), and ultimately a unicorn. Concepts like DevOps, Observability, Serverless, Pipelines, Testing, Lean Product, Agility, Experimentation, Team Topologies, ‘the right amounts of communication’, cognitive load. We don’t take these concepts, methodologies and practices as a mantra. We use them as a tool to achieve our goal: iterate our software product at pace, collaboratively. 

Finally, the thing that underpins my two previous favourites: the people. I genuinely like so many people here, and learn from them every day. That is something unique – at least in my experience.

Greg – The positivity. Everyone here is so proactive and “can-do”. It gives the feeling of innovativeness and excitement to problem solving.

There are lots of things I like though. For me the big “pull” to come and work at cinch was the tech. It’s “Serverless First”, heavily utilises AWS CDK, GitHub Actions and DataDog – all things I’m really interested in and find exciting to use. Most importantly on the tech-side, these technologies are used because they are the best tool for the job, not because they are “cool”.

Also worth a mention is the opportunity to learn. I think particularly with my role, it’s impossible to be an expert in every aspect of the work I do, which means there’s always so much to learn. Everyone I have met has been really supportive and willing to help.

How do you manage your time during your week to allow for upskilling/training time?

Toli – That is a very good question. Everyone learns in different ways. And this is not a cop out. I learn by doing. I learn by reading books and articles, watching videos, listening to podcasts, speaking to people internally, speaking to people externally, and being active in slack spaces. If I need to take some time out to learn something, I make some time for it. Otherwise, I am constantly learning and upskilling. Proof of this is that in my time at cinch I have upskilled and trained myself in a number of things: Azure infrastructure concepts and tooling, AWS infrastructure concepts & tooling, two pipeline providers (Azure DevOps, GitHub Actions), observability (concepts & tooling), reliability, incident management, serverless concepts, AWS services, TypeScript etc.

My current priority list for training is creating and facilitating workshops, public speaking, writing, AWS certs, and Wardley mapping.

Greg – As I mentioned above, I’m always learning at cinch. I know that a lot of people here like to carve out set time for learning, maybe a specific afternoon each week, for example. I think this works well if you’re working in a squad because it means everyone is learning at the same time, so you don’t get pressures or interruptions to learning time. But for me I am learning through just “doing my job”.

For the most part my experience so far has been to learn a new thing from someone, realise that another squad I work with could benefit from it, and go and share. But I’ve also enjoyed being part of ‘working groups’ – these are multi-discipline, cross-squad initiatives that run for a short time to solve one specific problem. A working group is established to solve a specific, tangible issue that we have at cinch and it’s down to the group to research, ideate and iterate to figure it out. These groups are really interesting places and it’s a good chance to work with some people outside of my usual circles.

Finally, there are a lot of people here who are very knowledgeable on a range of specific things and I’ve learnt a lot just from working with them and asking them questions.

How does a scaleup like cinch manage to maintain its culture and keep its people happy as processes and procedures have increased?

Toli – That is a very difficult question to answer. And the answer is slightly existential to cinch. cinch started out with a very strong and solid unwritten engineering culture due to the clear vision by its leadership. The problem domain was ripe and the approach taken was ideal: hire talented people, apply lean, good product engineering principles, and let them fly! Take away the faff from the teams. Help them excel. That has taken us a long way. And almost got us to where we are now. All this time (almost 2.5 years), we have been scaling – in terms of people and traffic/sales – rapidly. We have not had time to settle. And we did most of this almost entirely remotely. Naturally, as we became a scale up it’s important to make sure the culture we’ve created doesn’t get lost.

From what I can tell, we are doubling down on collaboration and applying good team topology and community techniques. We are starting and growing Communities of Practice and Communities of Interest so that we harvest the knowledge, expertise, and drive of the people we have hired. We are re-organising based on domain and teams’ cognitive overload. We are going back to the office as part of hybrid working to collaborate. Not to put bums on seats. If you go to the office, it’s usually laptops closed and white boards, sticky notes, games, ice breakers, and workshops everywhere. We are trialling modern survey techniques to get a happiness pulse (it’s actually called ‘Friday Pulse’). Somehow, at cinch, this type of thing doesn’t feel like corporate dullness. 

In parallel, on the engineering side, we are building an Engineering Progression framework, which includes Engineering Principles. We have built an internal ‘tech blueprint’ where we can surface patterns, examples, templates, principles, golden paths, tech radars. Anything that underpins an engineering culture. 

We are moving from being extremely synchronous to more asynchronous – we used to spend so much time with each other. At the scale we are at now, that’s not possible. So the challenge ahead is how we can make knowledge available to the right people at the right time. Internal blog posts, videos, API and event schemas etc.

Having spent almost 2 years working remotely, being able to have an office, and meeting people in person to develop and evolve relationships might be the catalyst to preserving our culture.

Greg – For me it all depends on what the original culture is like. From what I have learnt about the early days at cinch, about the approach and attitude, the culture started in an excellent place. A few key principles supported the culture then, and continue to support the culture now. Most importantly, those principles scale well. From my perspective those principles are:

• True Autonomy. Each squad is empowered to make their own decisions about almost everything, from tech to ways of working.
• Move fast and learn. We’re here to innovate and that means trying new things and sometimes what we try won’t be successful. And that’s ok! We learn from the mistakes and move on to the next thing.
• Avoid sentimentality. The company is so young and yet there are parts of it that have already been completely re-written 2 or even 3 times! Why? Because it was the right thing to do. This means that we aren’t lumbered with “legacy” systems and lots of tech debt which are just no fun to have to work on and maintain.

These principles foster a culture of mutual respect and trust. They promote innovativeness and dynamism over stagnation. They lead to a healthy tech stack and product which is fun to work on.

How does cinch support you professionally and personally to ensure you bring your best self to work every day?

Toli – I will start with professionally: I feel empowered every day to make an impact. I feel supported by knowing that if I need something I will get it. A book, books for people (I’m running a DevOps Handbook book club), tech equipment, training courses and time. As an example, I was giving a talk at a meetup recently. My line manager, Andy Norton – our Head of Engineering Practices – sent me a book called ‘Everything I know about life I learned from PowerPoint’ to support me (it’s a brilliant book, don’t judge from the title). What’s more, I have moved through roles at a pace I would not get elsewhere. 

Personally: whenever I have needed something I have had the space and time that I needed. Whether that is flexibility to spend a couple of hours at lunch time exercising and having lunch. Or whether that is needing some time off (and mental health support) for a recent immediate family issue I had. Whenever I needed a chat, I was able to chat with people. I have never felt I can’t reach out.

Greg – I think Toli has summed this up perfectly and my experience so far echoes his. Professionally speaking I think cinch is the most supportive place I have worked. To the extent that I almost feel like it’s my job to learn as much as possible! From a personal point of view I have always been treated really fairly and with respect. My manager, Jonnie, genuinely cares about me and everyone in our team and puts our health, wellbeing and happiness at the top of his priorities, which is such a reassuring thing.