Ash Grennan

Arise: Lessons from Solo Levelling for Software Engineers

Fri, 16 May 2025 00:00:00 +0000

In this anime, the protagonist Sung Jin-woo starts by fighting monsters. Initially, he’s weak relative to his peers. However, he receives a unique opportunity to grow stronger, gradually becoming an exceptional fighter. Eventually, he gains the ability to raise defeated enemies as Shadows, who fight alongside him. These Shadows fight autonomously, allowing him to step back. Is this where his strength stagnates? No, it accelerates.

In software engineering, we can use LLM-based tooling to augment ourselves. This has been the case for some time. Many of us tried these tools in their infancy and decided they were more of a hindrance than a help, an opinion I shared. With the advancements in tools like Cursor AI and refined language models, should we continue to dismiss their usefulness and not re-evaluate?

As Sung encounters more challenging dungeons, using his Shadow army becomes more valuable. From his new capability, Sung learns new skills, such as instructing and strategising. Just like an engineer who gains time by leveraging gen AI to handle the mundane, Sung’s reliance on his Shadow army becomes an enabler for personal growth.

The world is speeding up, and the amount of information thrust into our view is accelerating. Time spent reading, comprehending, and digesting articles in the era of digital information is on the decline and will continue to fall. Why, you ask? Because there’s a limit to how fast we can absorb information, and there’s already too much. There’s so much of it that our default reading style is skimming through information, often missing nuance. We are trying to drink from a firehose each day. We’re burnt out, not from overworking, but from overstimulation.

The next layer of abstraction

Programming has drastically changed over the past 70 years, from punch cards to assembly to OOP languages. Most developments are an abstraction over the previous instructions, and I’m sure we’re all happy about not having to use punch cards, waiting hours to retest our logic. However, over this period, there were many who shunned the changes that got us to this point; they resisted and rebelled against it.

When our worldview is challenged by something that we believe impacts our value addition, we choose ignorance. We measure our productivity by keystrokes, not by the quality of our ideas or the creativity behind our solutions. We’ve been conditioned this way, but original thought and novel ideas are what true productivity really is.

We have an opportunity to use the next layer of abstraction, just like a better paintbrush aiding a painter. It’s our responsibility to leverage our expertise and apply technology selectively, ensuring we understand the complexities and unknowns.

It’s by taking on difficult and unclear problems that we sharpen our internal tool, our mind. Using the cognitive bandwidth freed by advancements in tooling, we are more able to apply ourselves fully, like Sung Jin-woo. Do you need to type every key stroke? Doesn’t code get optimised by a compiler?

Sure, I hear you, why not slow down? That’s the right solution surely. This isn’t in our control to dictate. Though the pressure to move faster may feel inevitable in a competitive environment, intelligently applying these tools may offer us the cognitive breathing room to slow down our thinking process, focusing on deeper insights while maintaining output.

If you ship faster, you in turn learn faster and with this competitors will always be at least one step behind. They’ll never be able to catch up if done correctly.

Conclusion: a history lesson

In our control is the ability to squeeze the world to fit our cognitive bandwidth; tools like LLMs can greatly help with this. Socrates worried that writing things down would cause one to have a bad memory, yet it’s empowered humanity’s collective memory to learn. Similarly, LLMs, if used thoughtfully, won’t weaken our skills as engineers; they’ll amplify our ability to manage complexity and innovate. Wisdom isn’t fearing change and what that may take from us, but instead embracing what positives it may unlock within us.

Handling Zendesk events via EventBridge

Wed, 02 Oct 2024 00:00:00 +0000

Note: This article was written before the announcement on Zendesk deprecating their partnered EventBridge integration in December which can be replaced via a webhook integrating into EventBridge. However, this article is relevant to any event-based broker integration and the architecture discussions it entails are valuable.

Introduction

In this post, we will cover how we solved some inefficiencies in our business by using Zendesk’s EventBridge partnered connector in conjunction with an event-driven architecture. If you’re unfamiliar with Zendesk it’s a powerful customer service platform that helps organisations manage and resolve customer inquiries efficiently. We use this here at Moonpig for customer service operations.

AWS EventBridge is a serverless bus that enables services to be loosely coupled and event-driven. It can consume data from multiple sources and publish these to different destinations.

Problem Space

Within Moonpig when a customer interacts with us through our Chatbot via Zendesk we’re aware due to processes such as looking up an order. We are also aware of when a resolution has been reached. When this happens, we add meta information to each ticket, saving our support team time. One of the challenges we face is that Zendesk has a variable time window for ticket creation, so we don’t know when tickets will be available for our enrichment process.

Initially, this was solved via long polling, a common but inefficient pattern that can lead to increased infrastructure costs and higher API rate limits. Polling regularly consumes unnecessary resources and creates latency in data processing. However, thanks to the introduction of Zendesk’s partner resource, we can now move to a more efficient, event-driven process, which eliminates these challenges.

Architecture

For our architecture, we’ll be using AWS serverless offerings to create a redundant, event-driven architecture. Here’s our starting point:

At a high level, we have:

EventBridge is our bus, where the Zendesk bridge is connected and enables data to flow into our account. We’ll create rules that can generate filters and even transformations based on specific schema patterns.
SNS broadcasts notifications of business domain events, this enables a pub/sub setup that produces events to which other services can subscribe.
SQS > Lambda gives us a decoupled approach, giving a host of benefits such as redundancy, high availability and requiring no changes to any producer should this pattern change in any manner. Within our Lambda we can query other services for additional data.

Our architecture for this example looks good, however, we’re only dealing with two CRUD events on a single resource which is Ticket. But, what if we want more? There are over 100 different events across all resources, would our above architecture cope? The answer is yes, but that doesn’t mean it’s necessarily the best solution.

Being pragmatic we want to focus on value delivery and get the smallest unit out quickly to gain knowledge, however, we need to understand the tradeoffs if we’re to stick with the architecture in the context of an enterprise which follows the AWS Well-Architected Framework.

Considerations

Although decoupled, the proposed architecture still presents a level of rigidity because we need to manually update SNS subscriptions for each consumer. This kind of tight coupling can become an anti-pattern in event-driven architectures, as it reduces agility and increases maintenance overhead.
The second drawback is that this doesn’t empower our downstream consumers with granularity. We could use Amazon EventBridge’s cross-account event delivery feature. This approach would allow you to decouple your event producers and consumers even further, reducing the need to manage subscriptions directly on SNS topics.

EB to EB

Forwarding broad entity events into another EventBridge would allow for the following:

✅ Decoupling: Cross-account EventBridge removes the tight coupling of SNS subscriptions and allows better separation of concerns.
✅ Simplified Management: Events are routed directly to the target accounts’ EventBridge, simplifying the event routing logic.
✅ Granular Control: Each target account can define its own rules and targets, providing more granular control over event processing. This is impactful as it could save significant costs via our consumers not needing to act on irrelevant data.

A simple example of the power of this architecture would be a service only cares about an entity that has specific attributes, e.g. in the context of a ticket perhaps our service only wishes to ingest if the tags on a ticket are either foo or bar. With this architecture, we can use our own rules to make this possible and eliminate our consumers from receiving irrelevant data.

However, the architecture is crucial here, if we applied the above to our architecture in Figure 1 via highly specific event patterns, this could become an anti-pattern, making the system difficult to maintain and debug. It’s important to balance granularity with simplicity, ensuring that patterns are easy to manage over time.

Implementation

To create this setup we’d use OpenTofu, our preferred IaC tool. To keep this agnostic we’ll go through the AWS console instead. This section assumes the following:

EventBridge resource has been created and successfully connected as a partnered resource and information is available, there is a guide here.
You’ve created an SNS, or supported resource which can connect with an EB rule.

Steps

EventBridge > Rules > Create a rule, and call it ticket-created
Event Source > Select ‘Other’
We can ignore the Sample event section. However, this is useful for finding pre-existing patterns and testing our event pattern against specific data shapes.
In the Event Pattern section, we’ll need to specify what event we wish to receive, however, an important consideration is depending on your configuration of Zendesk workflows, the data you require may not be immediately available, which, because this is event-driven, would mean we’re missing data. For us, the event of a status change is appropriate so we’ll use this.

Support Ticket: Custom Status Changed

This means our example will look like the following in Event Pattern:

{
  "detail-type": ["Support Ticket: Custom Status Changed"],
  "resources": ["Support Ticket"]
}

You’ll also want to make sure Prefix matching is set in the dropdown.
Next, we’ll want to set a target for the data which matches the above rule, we’re going to choose our SNS, but this could easily be Kinesis, Lambda or another EventBridge
The SNS will now publish events based upon our resource and detail ruling, it’s important we test these rules to ensure our event contains the data we require, as it will be a snapshot of the entity at the time of changing

🤔 Conclusion

We’ve gone through setting up an event-driven system, from consuming SaaS data into our estate for internal consumption and speaking about the benefits of this being async and event-driven. We’ve also considered our EventBridge broker in the context of our AWS strategy and how it may be built upon, speaking about different architecture decisions we could take and how they would set us up for future success. Finally, we briefly touched on the high-level implementation using the AWS Console.

Reimagining Scrum Ceremonies

Mon, 25 Mar 2024 00:00:00 +0000

Effective Scrum ceremonies — Part 3

So far, we’ve covered the wins and drawbacks of communication methods such as writing documents and meetings. Next, we spoke about the heart of async, written words! We dove into using the golden circle to help us stick with recording communications to facilitate discussion and gain goodies such as inclusivity, thoughtfulness and flexibility.

Ceremonies

If you’re following Scrum, you’ll have a series of ceremonies that repeat on a weekly, or bi-weekly basis. An example of ceremonies for a 2-week sprint in a medium-sized team:

Standups (15 min) × 10
Sprint planning (3 hr) × 1
Sprint Review (2 hr) × 1
Sprint Retrospective (1.5 hr) × 1

That adds up to 9 hours, just over a day, every two weeks for each member of your team. Against a 38-hour working week (76 hours per sprint), that’s roughly 12% of 2 weeks on recurring meetings, and it’s pretty typical for these to run over too. We can agree this is a significant amount of time; it’s therefore worth exploring how we could improve on them from an efficiency standpoint.

Standup Meetings

One of the most adopted techniques in Extreme Programming, standup meetings generally include:

What we did yesterday
What we’re doing today
Blockers (if any)

With distributed people and teams, the standup can and does suffer. It’s quite easy for a standup to run over when everyone is sat down, and to become an environment to fixate on nuanced issues, derailing its purpose. With this, engagement can be impacted, we get into a we have the meeting because we have the meeting mindset as its original value and purpose erode.

Focusing on the numbered list above, here are some methods of improvement:

✅ Write updates — Using tooling effectively, points 1 & 2 can be answered by the board. If they can’t, it’s either a tooling problem or a human one. Team members should update via writing comments on stories and moving tickets into lanes / statuses. What happens when someone is out of the office? Unable to make the standup? Relying on sync communication creates frustration and slows us down.
✅ Flag stories — If you have a blocker, you’re likely already tackling and trying to resolve, not waiting until the next standup! Flag the ticket if blocked, write a comment explaining why, default to action, waiting to do this sync slows everything down.

Actioning the above points, it can become apparent the daily standup isn’t required. However, the key benefit of these meetings is the connection opportunity they provide. If your team still want them, remove the status update. Discuss the next actions, sprint goals, health check, high risk branches or changes.

There’s also a benefit of psychological safety via actioning the above, as the ticket communicates the progress of the story, avoiding lots of questions from multiple people.

Sprint Retrospective

The agile process is all about enabling continuous improvement. There’s no better way than having a retrospective! It’s a way to surface issues, celebrate success and enable actions to be taken away for future improvements. However, run poorly, they can do more harm than good.

😕 If they consistently don’t yield meaningful output, people lose faith in the ceremony
🔄 We all have a recency bias, recording events that recently happened, leaving more historic issues to happen again
⏱️ Running out of time, due to poor grouping, lack of voting, or absence of a theme

All these issues can be mostly solved by… you guessed it, async means. Here’s how we’ll make our retros more effective, efficient, and purposeful.

✅ Capture inputs — Enable and encourage the habit of everyone writing topics for the next retro throughout the sprint, these could be in a Google Form, doc, or even in their favourite note taking app. These can be sent to the facilitator of the retro if anonymity is not a concern.

✅ Group inputs — The host, ahead of time, groups related topics. If this was a Miro board then sticky notes in clusters achieves this. This helps with point 2.

✅ Voting — This enables focus and prioritises the topics people feel strongly about. This helps with point 3. As this is usually a quick process, it can be done before or during the meeting.

Sprint Planning

It’s a heavyweight ceremony and, unlike others, it involves many different subprocesses. Discussing these would require an article of its own. So, let’s discuss this purely from an async, efficiency perspective. Here’s how we can streamline our planning sessions.

Have some form of async backlog refining where time is set aside to enhance stories via metrics or more detail, ready for the sprint.
Tightly prioritise the backlog for the function of increasing focus around outcomes.
Define the sprint goals before the meeting, the Product Owner / Engineering Manager can draft this, it will give natural shape to the start of the meeting. Make this known before the meeting to all.
Encourage team members to look through the draft of stories to bring into the sprint async, here there’s the opportunity to write comments in the stories for clarification.
Have the meeting.

Doing the above will yield the following:

✅ Allows the team to think deeply about stories, thus creating better outcomes such as highlighting potential issues.
✅ We’ll have a list of questions already available to discuss during the meeting, this is huge.
✅ Reduce risk that you’re pursuing the wrong thing.
✅ We’ll already have thought about the sprint goal and seen the approximate stories, which will give a better confidence indicator on if the goal is achievable, or requires reduction.

Conclusion

We’ve discussed simplifying each ceremony with straightforward, asynchronous actions. As a result, we’re creating better fuel to accelerate meeting outcomes, ensuring we get the best out of each sync interaction. Furthermore, it enables contributions when it suits the individual and unlocks thoughtfulness, which can create better, faster delivery and greater working software.

The synchronous analysis of meetings

Fri, 08 Mar 2024 00:00:00 +0000

Balancing sync and async communication in the remote work era — Part 1

Let’s set the stage. In a distributed work environment, getting the balance between synchronous and asynchronous communication is difficult. This post discusses my experience during my career in tech around the nuances of both sync/async communication forms, exploring their benefits, trade-offs, and stating strategies to help optimise meetings.

Sync communication

✅ Fast nourishment
✅ Perishable
✅ Easily digestible (for most)

Async communication

✅ Harder to produce
✅ Strong and potent
✅ Lasts a very long time

A recap

But we’re already using asynchronous communications and tooling

Communication isn’t generally benchmarked in companies and their application and efficiency are not known. Businesses are often so focused on delivering value for customers or stakeholders that the tools we use to do that aren’t scrutinised.

Many of us thought, and correctly believed for a short time, that working remotely would enable us to concentrate, no noise or interruptions we thought! What in reality happened was that the noise and interruptions followed in the form of meetings. It’s a behavioural problem, it’s us.

Let’s recap the tradeoffs and benefits for meetings. This isn’t an exhaustive list but it’s good to remind ourselves of some:

👌 Easy: This isn’t a direct flaw, it’s a culture / behaviour one, but it’s worth restating. In a distributed company they are the easiest communication option, making them prone to misuse.
️ 🗣 Loudness: In decisions we want the best idea to win, not the loudest voice, with the nature of meetings that’s a hard thing to achieve, not everyone wants to voice their opinions due to nature, language, or simply needing to articulate, it’s not conducive to being inclusive.
💸 Can be expensive: Does the cost of the time the meeting requires create enough value to justify itself? There’s also opportunity costs here too, is this delaying you from working on something with higher value?
🚀 Quick decisions: The quality of our decisions depends upon the quality of our thoughts. When we’re optimising for fast thinking via communicating in a synchronous manner, it’s not possible to ponder and think deeply, we use our experience which is biased by what we’ve done before.
👥 Inclusivity: For people working in a second language, following a fast conversation means translating what others say while also assembling their own thoughts in a language they’ve adopted. I have been told this is incredibly difficult.

A few benefits for meetings include:

🔗 Connection: Working for extended periods in isolation whilst offering focus can be at the detriment of why we’re doing what we’re doing in the first place. Meetings remind us of why, our mission, and wanting to support and achieve outcomes as a collective.
🎯 Focusing on now: If something blows up, or a critical decision needs to be made, meetings make an excellent tool to agree a course of action efficiently.

There’s one other large area which has been intentionally missed because it can be both a strength and detriment which is breadth / depth.

Meeting etiquette — evolve meetings

We could immediately descend into async ways of communication and move some of our existing communications to leverage async, however there’s an issue that extends past simply taking action.

Adopting a different communication style isn’t something that’s easy, we need strong relationships in the team, understanding and agreement in ways of working, resolving ceremonies, having conversations with stakeholders and the list goes on. Instead, let’s look at first improving what’s already there.

❌ No agenda, no attenda: If you get invited to a meeting which doesn’t have an agenda, this would imply that the objectives and outcomes for this meeting haven’t been thought about. Sure, the creator could have forgotten, but encouraging the habit by asking for an agenda forces the exercise of thinking about the meeting’s purpose.
🍲 Preparation: Share any artifacts such as presentations, documents or video. Rather than the first half of the meeting being trying to understand the problem or scenario, instead it can start with clarifying questions, enabling better, faster output.
📚 Start with information: If people aren’t used to reading documents before meetings, Jeff Bezos has spoken about a technique, referring to it as “The smartest thing we ever did at Amazon”. This involves meeting attendees spending the first 5–10 minutes of each meeting reading a well prepared memo, everyone in the meeting does this and then the verbal part of the meetings begins.
⏺️ Allow thought recording: Allowing people to write their thoughts in a collaborative document can help those who otherwise wouldn’t contribute to the discussion. There are incredible benefits for such a simple action, greater inclusion and stronger decision output due to a greater collective.
📃 Document the outcomes: This is a no-brainer: not only does this have all the async benefits like write once, read by many but it’s incredibly important to give people context for moving forward on a decision if they’re unable to attend. It’s more efficient than recording a meeting unless a verbatim record of what happened is required.

You may have noticed something about these points, each one leverages async communications in some manner, we’re addressing some of the weaknesses of sync by applying async tools and patterns.

The scheduled meetings

What I’ve observed happening over time is the boiling frog. You’ll be involved in initiatives over your career which lead to scheduled meetings. This slowly reduces the amount of hours available in your working week and before you know it, a third of your working hours are eaten by scheduled meetings! You’re metaphorically boiling alive without realising it, as the number of working hours available for tasks gradually decreases.

It’s an outdated viewpoint that being in meetings therefore means you’re busy, thus in turn being productive. So, how to weed our calendars? Here’s what I’ve found useful:

📰 Importance of an agenda: We’ve already covered this above, what exactly is the meeting’s purpose? Is it just transferring information? Well, that sounds like documenting would be a great fit here or a tool to post and distribute status updates.
🙆 Participation: How often do you actively participate in the meeting? Are you just attending the meeting because it’s in your calendar and that’s what you’re supposed to do?
👍 Feedback: If attendance is not mandated, are you passionate about the topic? Is it making you better? Give feedback to the meeting owner first if you’re thinking about removing it from your calendar.

Not all scheduled meetings are bad, far from it. However, we need to be objective again. Many long-running scheduled meetings lose their impact and deliver marginal value over time, it’s easy to diverge from the initial intent. Many companies do forms of meeting resets to combat this problem.

Next time ⏭️

We’ll be exploring what async looks like for ceremonies and some of the issues around ‘async’ tooling!

Unlock async communication for engineering teams — Part 2

Fri, 08 Mar 2024 00:00:00 +0000

In this post, we’ll address the ‘bus factor’ and practical steps for creating asynchronous habits. Finally, we’ll be walking the path of creating a value system for async working by adopting a framework to aid in defining why we want to do this and most importantly, create some sticking power.

Encouraging behaviours, why async

Let us begin with an example of the bus factor. How many people leaving your team would impact value delivery? How many would halt support capabilities? This can be a sobering exercise. Sure, teams adapt to change, but wouldn’t it be nice if no additional cognitive load were placed on colleagues during such a loss? Wouldn’t it be ideal if velocity didn’t drop and your team were just as capable when dealing with incidents?

This is one of an abundant pool of reasons why async for information capture and delivery is great. The higher the velocity an organisation has, the less long-term memory it has by that very nature. Think of this as a sliding window moving forward, but not expanding. We have a remedy for such a symptom: write.

Consider the wisdom of former HP CEO Lew Platt who encapsulates the point beautifully:

If only HP knew what HP knows

Taking Action

Previously, we spoke about enhancing meetings via injecting async tools and processes. In this section we’ll touch on how to be a force for async, getting started, and building the habits that make this a part of your daily toolset.

1) Starting with why

You can be motivated and have the tooling to start async, but none of this will yield value without buy-in from the right people, your team members. Without this, you’re putting the cart before the horse. It’s critical that this happens and everyone sees the value.

To gain buy-in for capturing information which will enable async, we can use Simon Sinek’s Golden Circle model by first defining a ‘Why’ to communicate the purpose and benefits.

A brief example of this for our purposes:

Why (The Purpose) — This is the reason you’re doing this; it has to mean something for everyone involved. Examples could be, increasing inclusivity, reducing repetition, decreasing knowledge silos, it shouldn’t be just a procedure (i.e. something you do), but justify something strategic too (a mission to complete). The company behind WordPress (Automattic) uses both async and the Golden Circle method, their Why is to enable a global talent pool and maintain continuous productivity
How (The Process) — Highlighting tools and how they’ll be used to enable the Why
What (The Outcome) — What will the outcomes be, this may be reduction in meeting times, increased innovation, this should form a single sentence, and have a few lasting, purposeful outcomes. Without outcomes that can be discussed and measured, you won’t have a method of knowing if what you’re doing is actually working

2) Get good at writing

The number one tool for transferring information is written words. In the right medium, a digital document can scale to be read by millions of people over months or years. It can be versioned, edited, revisited, translated, searched and made accessible around the globe in seconds.

I want to highlight that writing is hard. But, it’s the unlocker for making async work. Just like starting at the gym after a long hiatus, it may be difficult at first. But you will get better, it will get easier. If that’s not enough, think about your Why, your purpose in doing this. Is it to empower people who are neurodiverse, non-native English speakers, introverted? Remind yourself of Why when you don’t want to write.

Writing or recording enables more people to be involved in the conversations, capturing perspectives we would miss otherwise. It’s not just writing too, media recordings get a noteworthy mention as a medium that captures non-verbal cues, tonality, and a demonstration of ideas. Whilst writing gets first place, creating media to express information is still on the podium.

3) Kill the mouse

You’re in a meeting and fortunate enough to have two screens, one screen has your meeting, the other Google Docs etc. Unless the session is interactive, you don’t need your mouse. It’s a navigator and we don’t need to navigate. All it will do is welcome distraction via checking email, messages, that unit test that won’t pass. Put it out of arm’s reach.

4) Create a hub of information

We need to put this information somewhere, naturally, this depends on tooling options available. What is important though, is agreeing where stuff will live, and how you’ll structure it (How). Some examples include:

Meeting minutes, the outcomes. Initially, avoid using AI if viable because you won’t have a benchmark for comparison. Afterwards, assess whether tools capture key points and action items more accurately. Are they searchable and summarised? From there your team can make an informed decision
ADR (Architectural Decision Records), often seen as a challenge for engineering teams to maintain, they give context and save a tremendous amount of time reading about the evolution of a service. Experiment, find a system that works for your team, perhaps adding an ADR into the projects directory doesn’t work, and that’s ok, you can just link to the ADRs in your project’s README
Stakeholder decisions aka business decisions. This is a no brainer, if you don’t have written down what you’re aiming for and agreed timelines, you’re going to get into trouble and create unwarranted pressure. Ever played the Telephone Game? You’re unwittingly playing this if things aren’t recorded
PRs (if applicable), Placing links to these alongside all the information above creates an effective audit trail, seeing how a change happened relative to a business decision change
Ideas, Have a great idea? Write it down! Put it in your information hub, enabling your team to review in their own time, the more complex the idea, the more difficult to discuss synchronously, enable thoughtfulness around the idea first before verbally discussing

There’s more to talk about here, but this is a great start, use introspection to decipher what you need to know first, define simple templates, make recording information easy and frictionless.

5) A Mantra

Once you’ve decided what to record (starting small can help) next is actually taking action, we’ve spoken about how writing is hard; this is the remembering to record part. Here are some useful tips on ensuring this happens:

Who’s writing — In a meeting / discussion, ask at the start who’s capturing the information? Don’t write this in your Notepad or in a medium that’s local to you, always record in the tool you’ve agreed, else you risk forgetting to transfer
Repeat — If warranted, at the end of a discussion, the writer should recap what they’ve written in a summarised manner, this may take 30 seconds, and not only is it useful to ensure information is correct but gives everyone a high level recap
Post it somewhere — If you’ve created a new document and added it to your information hub, consider also posting in a dedicated IM channel for async updates, this gives a timeline of discussion and recordings

6) Respond with a link

Next on building your knowledge hub: if someone asks for information that you have to explain, regardless of communication medium, write it down and send a link. If you commit to doing this, over time you’ll have more context, history and time-saving capabilities. It’s an investment that pays dividends.

There’s also the added win that people don’t receive a slightly different interpretation or recollection of your thoughts and ideas each time you state them.

Conclusion

The path to moving async isn’t easy to follow, however, defining our purpose and commitment to change means we’re starting off strong. We humans are imitation machines, so vote with your actions and people will follow. It’s hard to start, but you’ll be creating a better present and future too.

Modern Software Engineering Book Reflections

Tue, 28 Nov 2023 00:00:00 +0000

Recently, I read Dave Farley’s Modern Software Engineering publication which outlines a modern, sensible approach to engineering. It articulately describes what it means to be an engineer, and how to tackle engineering challenges.

Dave’s thought process and views on many topics felt familiar and similar to my own, not because we’ve worked together, but because I have the scars of trying many things that didn’t work. Reading each chapter was akin to catching up with an old colleague who’s been there, has seen what works and what does not. It was a refreshing read that I highly recommend.

Reflecting on Dave’s great work, I decided to write my own views after being inspired by Dave, and other thought leaders in our industry.

⚙️ Building Incrementally

Performing heavy upfront analysis and design slows progress; in simple software this may be OK. However, in the current era of building scalable, reliable and fault-tolerant systems, simple is rare. Complex systems aren’t produced by a single mind on the first iteration; instead they are built incrementally, by learning what works.

Some ideas which looked great on paper don’t satisfy requirements because of unexpected unknowns. If you’ve ever worked using waterfall methodology this was its biggest flaw, with many notable systems turning into a big ball of mud, despite hundreds of millions of investment.

We’re in a unique position as engineers, because there are many disciplines that can’t just pivot like us, for example, take building a bridge that spans several miles: a change of plan may be extremely difficult, and could be impossible due to constraints. As software engineers, we can delete code and remove resources.

🔬 Performing Engineering

Being a software engineer is far more than just writing code. Getting architecture and design right for the component you’re working on is important, iteratively reflecting on approaches, learning what works, creating hypotheses, experimenting and drawing conclusions is the value proposition of a good engineer.

As engineers, we need to select approaches and be aware of what our choices really mean, what are the tradeoffs we’re making in our approach? A simple example would be considering CAP theorem. We are trying to organise complexity all the time. It may scale linearly but, ideally, not exponentially, and it is up to engineers to try and tame it.

Code is the output of good engineering, not the driver of it

There’s not great value in clever code, we want code to be simple, expressive, and easy to comprehend. In systems you’re building as part of a team, you want to use code as a communication tool, things that aren’t clear can lead to misunderstandings and therefore bugs.

Compilers are very good at finding optimisations, so whether something can be written in one or three lines shouldn’t be a concern of performance, and if in doubt, test!

🌻 Change

Change is a constant, therefore enabling change should be a first-class consideration in our builds. Following an agile approach of small, incremental, iterative change forces us to build in ways that adhere to principles we’ve already mentioned (modularity, SoC etc), the payoff from this is that it greatly helps to flatten the cost of change in our system, not only in its infancy, but across its entire lifespan.

As you’ve likely experienced, tightly coupled systems become more difficult, more expensive and more dangerous to change. Releasing software like this becomes a scheduled event; because of this, things take longer and releases are delayed. This is far from ideal: we’re slowing down the rate at which we can build, measure, and learn. Meanwhile a competitor that can release more often has a competitive advantage that compounds over time. They could learn faster, and accelerate past you.

So, we agree that change is important, and changes often require having confidence in our releases. If we are confident that each release is bug-free to the best of our knowledge, then the time it takes to release working software (compile, test) should be the only blocker. We can develop this confidence by following a test-driven approach (TDD); following this forces good practices into our design and code.

Furthermore, having a deterministic method of releasing software with automated testing that increases confidence and can exercise our code at different levels is only a good thing. A continuous integration/deployment system is the tool for this; we can include processes such as creating the binaries for our software, performing automated tests from unit to E2E, scanning for library vulnerabilities and so on.

An effective continuous integration process that runs in a timely manner is a strategy that enables some of the largest companies in the world to release thousands of times a day across hundreds of engineering teams.

🧚 Dogma

It can be easy for a team to fall into a trap of doing what they know, why? Well, quite simply it’s worked in the past, or it’s been seen in other projects. A trivial example of this could be installing half a dozen different ruling configurations. These rules dictate how we write specific idioms of code such as loops, types, order of function, documentation, and so on.

Whilst the intent is good, many of these dictator configs are pulled from large companies with specific goals, reasons and styling choices. Do these choices help our team in the specific context of what we’re trying to achieve? Are we as engineers in agreement of following these conventions because they’ll help us? It’s for us to decide, not to blindly follow what has been done before and assume everything applies to everyone.

“Have no respect whatsoever for authority; forget who said it and instead look what he starts with, where he ends up, and ask yourself, ‘Is it reasonable?’”

— Richard Feynman, What Do You Care What Other People Think? (1988)

It’s our responsibility as engineers to make decisions based upon data, this evidence helps us navigate the sea of possibilities and acts as our compass to guide towards a desired outcome. We can also evaluate if we are closer, or further away from our end goal too. Basing our technical decisions on who was the loudest, or had the most conviction, isn’t how we should work.

🏁 Conclusion

Software engineering is difficult, period. Whilst tooling and technology are ever advancing, so is the complexity of our software. We rely on it for mission-critical systems, financial markets, and piloting all manner of vehicles, and its capabilities are ever increasing. The expectation even for general systems such as ecommerce is that they should be available, fast and fault-free regardless of where they are accessed.

Things do go wrong, meeting expectations for customers, or complying with legislation, presents many engineering challenges, and we should recognise and build resilience and fault tolerance into what we do.

How we design systems is critical to facilitate change. Good engineering teams build great software on all frameworks and languages. However, it’s difficult to overcome the wrong design choices, especially if they’re cemented in. It is also not enough for our software to be modular; we need modular organisations too.

If your ability to release software quickly in order to learn is blocked by constraints of depending on other teams, you’re not agile. Being decoupled is a main metric of high-performing teams and organisations; you can only go as fast as your slowest constraint.

Finally, we need to internalise the fact that systems impact real people such as you and me. An error or mistake could have a devastating impact on a person’s life; it could lead to a refusal of a loan, visa, mortgage, healthcare and much more.

GitHub Actions — Supercharged

Wed, 18 Oct 2023 00:00:00 +0000

Having worked with various CI/CD platforms like Bitbucket, GitLab and AWS CodePipeline, I’ve come to deeply appreciate GitHub Actions. Its implementation, in my opinion, surpasses the rest.

Strengths of GitHub Actions:

♻️ Reusable workflows: This makes it easy to maintain and replicate processes across projects.
👾 Concise YAML syntax: For those who grumble about indentation, a linter can be your best friend.
🔁 Dedicated Repositories for Workflow Steps: Each step having its dedicated GitHub repository simplifies documentation searches and troubleshooting.
🤼 Community-Driven Workflows: The community’s involvement ensures continuous improvements and a wealth of shared knowledge.

Migrating to GitHub Actions, both for personal and professional projects, has been a mostly seamless experience. There were occasional bumps, like misplaced artifact locations, but the overall impression remains positive. So, where are the pitfalls?

The Challenges with GitHub Actions

Imagine assembling a flat-pack furniture item. The instructions are clear, and assembling the pieces feels intuitive. However, ensuring that you’ve used the right screw or bolt in the correct location can be a tedious process. To know if you did it right, someone would need to stress-test the assembled piece. Not the most efficient method, right?

Consider this GitHub Actions step:

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          ref: ${{ inputs.relase-tag }}

Did you spot the issue? Despite inputs containing a release-tag, it’s mistakenly referenced as relase-tag.

Result? A failed build 🤦.

Tools to the Rescue

There are some great tools out there that can help avoid the dreaded Process completed with exit code 1 error.

✨ 1) Actionlint

Actionlint is a robust static checker for GitHub Actions workflow files. It ensures your definitions are valid and sidesteps common pitfalls. Its features include:

Expression Checks: It ensures that GitHub Actions’ special expression syntax, ${{ }}, is correctly formed
Syntax Verification: Useful for identifying basic YAML syntax errors
Dependency Checks: It verifies your job dependencies to ensure that ‘needs’ references in a job correctly point to an existing job
Schema Validation: Confirms that events, jobs, and steps are correctly defined

Download the binary from its official repository and run actionlint in your workflow directory. Its swift analysis makes it an ideal addition to git hooks.

✨ 2) VS Code Extension — GitHub Actions:

This extension for VS Code enhances your GitHub Actions experience by:

Offering Autocomplete and IntelliSense for GitHub Actions YAML.
Providing an Intuitive Interface to visualize workflows, view run details, and get immediate status feedback.
Showcasing Inline Error and Warning Annotations directly in the editor.

✨ 3) nektos/act:

Want to emulate GitHub Actions on your local machine? nektos/act is your solution. It offers rapid feedback without the need for pushes, and for those mindful of costs, it’s a saver 💰.

The tool utilizes Docker containers to emulate GitHub Actions runners, ensuring environment consistency. However, bear in mind that sometimes these containers may lag behind the continually evolving GitHub ecosystem. But for experimentation specific to GHA? It’s an excellent tool which creates a rapid feedback loop. There’s a great guide to get running here.

Conclusion

Harnessing the power of GitHub Actions combined with these tools can streamline your CI/CD process. By preemptively catching errors and refining workflows, you’ll spend less time troubleshooting and more time deploying.

A Waste of Time

Mon, 28 Aug 2023 00:00:00 +0000

“That’s a waste of time.” We’ve all heard it, and most of us have said it.

Outside of collaboration, the application of the phrase has an air of contempt, disapproval or even disgust about activities or life choices that an individual is engaging in.

Many people believe they possess superior wisdom and, therefore, feel qualified to offer such advice. However, this is often not the case.

N.B. In the context of business when you’re trying to achieve an outcome, the phrase is perfectly valid and helps to create discussion around focusing on the right problem or trying to ensure value delivery against time. This post is about unwelcome opinions about how we spend our own time.

Example activities

🎮 Gaming is a waste of time
🏊‍♂️ Swimming (or, insert your chosen sport) is a waste of time
🎶 Creating music is a waste of time
🎭 Performing arts is a waste of time
🏫 That degree is a waste of time

No, you’re not more enlightened — the loudest voice in the room

If you were to challenge a person that said this to quantify what a ‘waste of time’ actually was, you’d get some answers in the region of:

Doesn’t have an output
Could be doing ‘better’ things with the time
Because you’re just sat there
There’s no money in it

This is all wrong of course, but where does this mindset come from? There’s a prevalent societal notion: first get educated, then land a job. That’s it. Therefore, if we aren’t working towards this, some activities are construed as ‘time wasting’ during our younger life.

The joy of chance

The only way you could define whether something was a waste of time would be to be an all-knowing deity who knew the beginnings and ends of everything for an individual and their choices. I’d argue even this is very tricky due to the butterfly effect, and negative outcomes have some positive ones.

Into Chaos Theory

Every micro decision we make shapes our life, for better or worse, but ultimately we can’t know that. Therefore, how could someone know an action can’t justify the time it takes? Taking time to do things that give us joy is not only the best thing for us, but can lead to making connections, relationships and all the things that make us human.

Ryan, a shy, somewhat introverted young man binge-watches a TV show by his favourite director
A few weeks later, there’s a conversation at a work lunch, and Ryan unusually jumps into it
There’s someone else who also adores the show and the director, from this a friendship is formed
Several months on Ryan is invited to his co-worker’s (now friend’s) BBQ, he gets talking to a girl…

The ‘waste of time’ of watching a TV show led Ryan to friendships and opportunities he wouldn’t have had otherwise.

In defence of

However, despite the criticism, there could be some justification here. One could argue that excessive time investment in a particular task is what they’re trying to highlight, but how do you define excessive? I think the simplest test is whether it comes at the detriment of everything else, or of activities that likely have equal or greater value than the one being pursued. Diminishing returns could also play a factor; think of the 80/20 rule.

Sometimes, putting most of your time into one thing is the best course, but as a default choice? Other things will suffer when it takes a higher percentage of the 24 hours we have in a day.

To conclude

Telling people they’re wasting their time is the wrong approach. If you wish to help, perhaps suggest ideas to nudge the individual into a less weighted time expenditure.

There’s quite an irony in writing this post, but not knowing where it will lead is the exact reason it should exist.

Lambda Power Tuning Case Study

Sat, 18 Jun 2022 00:00:00 +0000

Intro

There are two metrics for costing an AWS Lambda invocation, the memory size and running duration. Since we’re charged by GB-seconds in Lambda, the faster our lambda executes, the less we’ll be charged. Because of this, we could increase memory (and therefore more vCPU), yielding faster computation and therefore a decrease in execution time, and pay less. Interesting, right?

Measuring

Whilst an interesting problem, this is difficult to measure since we’ll need many invocations to produce something statistically significant, and then repeat this for many memory values. Fortunately for us, the lambda power tuning tool exists which invokes your AWS Lambda against different memory settings and fires n invocations and records the execution time using a state machine run as Step Functions.

From this data a visualization is generated displaying execution times and average cost of the configuration. Neat. An example can be seen below:

👉 How to interpret the chart above: execution time goes from 35s with 128MB to less than 3s with 1.5GB, while being 14% cheaper to run.

Scenario

We’ll test this against a real Lambda service used over at Moonpig. From digging around in logs I observed a specific non-I/O operation appeared quite slow and likely CPU-bound, a great case for the power tuning tool!

Note: I got too aggressive with cropping the above, the Y-axis is response times, you’ll see this on additional graphs, sorry!

Against 30 invocations per memory setting, the initial results were disappointing since increasing memory should yield something fairly linear, however, that’s not the case. After some digging, a third-party service had varied response times for each call, therefore, skewing the results.

Since the goal of this is to determine optimal CPU usage and I/O operations are not a factor, all I/O calls were removed to create a more deterministic computation, allowing us to focus on the specific code computation mentioned above. Any HTTP calls were mocked.

Round 2

This produced the desired result, with a much more linear progression, costs remaining similar throughout each configuration due to consistent time reductions.

Since the higher value memory performed best, a macro-level approach was taken, starting from the best result of 1536MB in the previous test and incrementally scaling up to 3072MB. The x-axis shows all values used.

Depending on optimization preference on cost vs performance, we can see 1792MB being the lowest cost. At 2240MB we save < 10ms, but the additional cost isn’t worth it for a service that isn’t latency-sensitive in this case.

Conclusion

From our deterministic setup, we’ve improved from 900ms to less than 100ms whilst fractionally reducing cost.

This is a time reduction of 88.8% 🔥 for the computation without much work. Given how easy it is to do this, I would encourage you to give it a try.

Setup Local Serverless Environment with npm packages

Tue, 04 Jan 2022 00:00:00 +0000

You have written some Serverless IaC and have created several Lambdas which need to either read or write to DynamoDB, the Lambdas are proxied through an API gateway. You want the ability to test your code locally whilst developing functionality and easily give other developers the same ability just by cloning and running the project.

You have different environments (beta/staging/production) and would like an env for local development to easily use specific configs.

An example of this all working can be found here 👉 https://github.com/ash-grennan/serverless-offline-with-dynamodb.

Plugins

For this we’re going to install several Serverless plugins, these provide additional functionality and cater for many common scenarios that may arise during development or simply enhancing IaC.

Serverless Offline — Used to locally run Lambdas and API Gateway, will provide an endpoint per Lambda and supports hot reloading during development.

Serverless DynamoDB Local — This will create an emulation of DynamoDB locally which is required for the Lambdas in this scenario.

It’s also worth mentioning that DynamoDB local will require JRE which can be found here.

Getting Started

First, you’ll want to install those plugins as dev dependencies, so:

npm i --save-dev serverless-offline serverless-dynamodb-local

Next, you’ll want to locate the plugins section (this is present in most Serverless templates) and add those plugins, it should look like this in TypeScript:

  service: "serverless-offline-with-dynamodb",
  frameworkVersion: "2",
  plugins: [
    "serverless-esbuild",
    "serverless-offline",
    "serverless-dynamodb-local",
  ],
  ...

Awesome, now we’ll run our Serverless offline plugin, we’ll pass in a stage environment since some of our IaC expects a stage variable such as our table.

serverless offline --stage=local

If the offline plugin has run the Serverless successfully, you’ll get a list of API GW endpoints similar to the below:

GET  | http://localhost:3000/local/api/customer/{id}
POST  | http://localhost:3000/local/api/customer

Now, if the Lambda invoked requires access to DynamoDB, you’ll receive an error. This is because we need to create our local db instance and point our calling code to the local address of this emulation, let’s do that now.

To do this, we’ll wrap DocumentClient in an abstraction and simply use this in any calling code, within this facade we’ll simply use a condition to change the DynamoDB location depending on whether it’s local.

const getClient = (stage: string) : DocumentClient => {
    if (stage == "local") {
        return new DocumentClient({
            region: 'localhost',
            endpoint: 'http://localhost:8000'
          })
    }
    return new DocumentClient({ region: "eu-west-2" });
}

export default getClient;

Side note: region should be from your config, we’re skipping some steps since it’s out of scope for this article

And then our calling code would look like:

const documentClient = createClient(process.env.ENVIRONMENT);

We can now run the following command:

serverless dynamodb start --stage=local

If everything has been run correctly, your Lambda will now be able to speak to your DynamoDB. We can look at our emulation via installing NoSQL Workbench for DynamoDB

Once installed, go Operation builder > + Add Connection and select DynamoDB local tab, it should look like the below:

Seed Data

Fortunately, it’s trivial to add seed data as part of our start command for DynamoDB, create a JSON file and add the JSON objects to an array:

[
    {
        "customerId": "ec3808fb-ecb7-48b6-8d82-67529ad0c6ef",
        "firstName": "Ash",
        "lastName": "Grennan"
    },
    ...
]

Next, we’ll want to define a DynamoDB config, in our scenario this will only be used for our local environment but could be extended to include additional configuration across many environments.

    seed: {
      local : {
        sources: {
          table: "local-customers",
          sources: ["./seed-data.json"]
        }
      }
    }

Then run the below:

serverless dynamodb start --migrate --stage=local

The table will now contain JSON objects which can be verified by NoSQL Workbench.

Conclusion

One of the strengths of this approach is that another developer can simply run 3 terminal commands and be running immediately whilst leveraging the existing IaC.

Naturally, there are drawbacks, for instance, Serverless offline cannot emulate everything, an example of this would be schema definitions for API GW models, in which case you’d want something more robust such as LocalStack, which has licensed versions providing even more functionality, such as IAM.

In case you missed above, here’s the GitHub link to the example 👊.

AWS Serverless FTP File Processing Service 👊

Tue, 28 Dec 2021 00:00:00 +0000

Introduction

In this article, we aim to tackle a common business task which is file processing 📂. If you’ve been in development for some time, it’s quite probable you’ve needed to grab a file from a resource (such as an FTP server), perform a read/write operation, and store the results in some form of managed data structure such as a DBMS.

From batch scripts to Windows Services 💻, there are dozens of ways to accomplish this. Let’s see how we can take a modern approach using AWS resources.

The Scenario

Acme Corporation 💥 need a file processing system. This needs to pick up CSV files from an FTP server and read the data, and each file requires an external API check on some fields. It’s been known for the CSV files to have issues in the past regarding their formatting, the system should flag files it cannot process for manual inspection. The system should continue to process other files regardless of individual file issues.

During the Christmas period 🎄 the company have an influx of files, being able to process more files than the average whilst maintaining the same time period is important.

Polling & Transfer Service

Let’s briefly go over the process flows available in this diagram as a ‘best effort’ goal of defining their purpose.

1. Lambda (File Fetch)

This is our poll service, its sole purpose is to connect to the FTP server on an interval, check a directory for files, download and send them to an S3 bucket.

✅ File Processing - There’s no processing of this file in here, the reason is our Lambda can only run for 15 minutes, we don’t want our Lambda stuck processing and downloading each file.

✅ Security - The above gives greater separation of concerns, which in turn enables us to have a refined execution policy with specific resource access.

2. CloudWatch

As the Lambda is used for file polling and downloading, the CloudWatch log data becomes more granular via just logging those I/O events. We can create some alarms which will alert users to issues via SNS or direct e-mail depending upon requirements.

There isn’t much to discuss, other than this is highly useful for event tracking and debugging.

3. S3 (File Bucket)

This acts as our temporary document store for our downloaded files and is a key part of our distributed, resilient file processing service. We will transfer our data over a VPC Endpoint to our Bucket, ensuring this doesn’t travel over the public Internet. There’s a good few advantages to storing the documents in our service, let’s first go over features of S3 we’re leveraging.

✅ Lifecycle Policy - We’ll use this to delete our files automatically to save cost. The retention may need to be agreed upon by the data provider or your company to consider PII when retaining the file. If there’s a processing issue we don’t need to re-download from the remote server.

This can be incredibly useful not only for saving time and operational overhead, but files can be short-lived or overwritten on FTP servers and therefore we avoid the headache of asking for files.

✅ Event Notification - We can hook into new object created events, allowing automatic integration with AWS SQS, queueing our file for future processing without writing any code.

Parsing & Processing Service

4. SQS

Our SQS allows for massive flexibility and resiliency and provides a mechanism for decoupling between our service architecture. Let’s outline some of the benefits here:

✅ Automatic Retry - If the Lambda fails to process a file, the entry in the queue will re-appear due to the visibility timeout mechanism. Processing files involves hitting an external API for validation, which could experience issues; in this case n retries is great to cope with random failure outside of the service’s control.

✅ DLQ - A dead-letter queue is incredibly useful for our scenario, we want to know if a file cannot be processed and most importantly we need to know which file in question needs to be inspected, it could be our code, or simply an issue with the file. Furthermore, since our message is moved to the DLQ after a finite number of processing attempts, we save time and money and avoid potentially blocking other files from being processed 👍.

✅ Elasticity - Currently one Lambda to process files is enough, however, if the velocity of files grows, event source scaling can horizontally scale n Lambdas based upon our queue length 👌.

5. Lambda

Here’s our file processor, its job is to read files, perform some data transformation and call a third-party service to validate attributes, once this has been completed the data is then stored in a DynamoDB table for other services outside of the scope of this problem to consume.

For this, we can use the S3 VPC Gateway endpoint to keep the file stream off the public Internet and purely in the AWS private network, eliminating S3 data transfer cost too.
We could use batching to save invocation cost via batching our SQS messages together, however, this would make debugging a file failure more troublesome and isn’t needed for our requirements
We can give our Lambda an explicit iamRole to access our specific DynamoDB table via its resource id, this allows us to easily follow least privilege and simply grant what we need
We can put a Lambda in our VPC and therefore restrict via Security Group to access required external services

6. SNS

The SNS sends a notification after processing, this could send an e-mail or log a successful process and if there’s a consistent baseline, have an alarm trigger below that threshold for a manual review.

Worthy Mentions

Since we don’t want to get too tied down with all the options available in regards to resource configuration, I’ve omitted what we don’t need to save the article from being exhaustive, here are a few things to consider for similar scenarios

👉 FIFO Queue - This guarantees order and prevents possible duplication of messages in an SQS queue, highly useful if duplication of messages cannot happen or order of messages is crucial

👉 S3 Cross Region - If this was a mission-critical system or our files could never be re-created, exploring S3 region replication would be useful as part of a disaster recovery plan

👉 Batch processing/size - Increasing the amount of messages included in a single Lambda invocation, this could make debugging individual files more difficult, but would result in a performance increase (due to lack of warmup time) and some cost saving from a time POV

👉 AWS Transfer Family - A little outside of scope, but if the FTP server were created using AWS Transfer Family, not only would this resource scale automatically, it would also provide workflows to transfer directly to S3 or EFS. In our scenario this would save work and reduce complexity.

Conclusion

That’s a wrap for the service, this scenario should be an approximation of a standard file processing system and its default expectations; i.e. it shouldn’t stop working if a single file cannot be processed, and the culprit should be logged etc. We’ve skipped much of the finer detail to avoid verbiage.

Beginner - Fluent Interfaces

Sun, 16 May 2021 00:00:00 +0000

So, let’s continue from our last post method chaining with C#!

To recap, Jim 🧔 found a cleaner, more efficient method of building his sandwich, but he’s discovered there should be a specific order to the ingredients.

With method chaining we can’t enforce a specific order of methods called. Let’s illustrate this problem.

    class SandwichFactory
    {

        public SandwichFactory AddBread()
        {
            // implementation omitted for brevity
            return this;
        }

        public SandwichFactory AddCheese()
        {
            // implementation omitted for brevity
            return this;
        }

    }

We, as sandwich construction experts, want to enforce the following pattern:

🍞 Bread
🧀 Cheese
🍞 Bread

However, a consumer of our SandwichFactory class could do the following:

    var createSandwich = new SandwichFactory()
        .AddCheese()
        .AddBread();

Or something worse! Let’s fix this problem.

Coming up with a nice fluent API requires a good bit of thought - Martin Fowler

Depending on the API we want to provide, complexity can rapidly escalate. For our example, we’re going to keep this small to illustrate the concepts more than application. Let’s provide our solution below and go through the reasoning behind decisions made in regards to fluent interfaces:

        public sealed class SandwichFactory : ISandwichBread, ISandwichCheese
        {
            private SandwichFactory()
            {
            }

            public static ISandwichBread Create => new SandwichFactory();

            public ISandwichCheese AddBread()
            {
                throw new NotImplementedException();
            }

            public ISandwichBread AddCheese()
            {
                throw new NotImplementedException();
            }
        }

        public interface ISandwichBread
        {
            ISandwichCheese AddBread();
        }

        public interface ISandwichCheese
        {
            ISandwichBread AddCheese();
        }

The constructor needs to be private - We don’t want a direct instance of our class to be created, since this would provide access to the AddBread and AddCheese methods (which is exactly what we’re trying to avoid)!
Static Entry point - our entry point, in this case Create should return the interface we want to start with which is ISandwichBread and an instance of our class
Keep parameters low - we don’t want to use tons of parameters per method, in our example we’re using 0, but it’s something to be mindful of

Consuming the SandwichFactory API, we now can’t add cheese 🧀 before bread 🍞, since our design enforces a bread > cheese hierarchy.

        SandwichFactory.Create
            .AddBread()
            .AddCheese()
            .AddBread();

Simples! 💪

Beginner - Method Chaining

Sun, 18 Apr 2021 00:00:00 +0000

I was talking to a recent bootcamp grad who had not covered this within their course syllabus, so let’s jump into an example.

Scenario

Jim 🧔 is making a sandwich. He already knows which of the ingredients in his fridge he’ll be using:

Mayonnaise ⬜
Lettuce 🍃
Cheese 🧀
Ham 🥓

Jim will need to visit the fridge for each ingredient added to the sandwich. Each trip requires a reference to the fridge, and removal of an item.

Could Jim make this sandwich creation operation better?

When we think about this scenario as code, it would be something like:

    var fridge = new Fridge();

    fridge.RemoveMayonnaise();
    fridge.RemoveCheese();
    fridge.RemoveHam();
    fridge.RemoveLettuce();

_{Figure 1: fridge and removal of items}

This pattern of calling multiple methods from an object is something very common in object-oriented programming. Whilst perfectly valid, this suffers from some issues:

We’re re-referencing the object, again and again, typing the word fridge and invoking a method each line
We’re creating a lot of noise and lines

Whilst clear in the example above, it lacks any form of relation, with each call being completely independent. Think of this as a series of cogs.

In our code example, the links are placed one after another, but they could become easily separated and it’s hard to keep them in order. Now imagine the links linking together like a… chain! We have a specific order; we can even enforce a certain order, but that’s a future topic 😉.

Let’s look at how method chaining can help us here.

If you’ve used anything from the System.Linq namespace, it’s highly likely you’ve used method chaining, an example of this can be found below:

            var people = peoples.Where(p => p.Age >= 30)
                .OrderByDescending(p => p.Name)
                .Take(10)
                .ToList();

_{Note: the example above uses fluent interfaces too, but we’ll
get into that another time.}

Some benefits of method chaining are:

To form a chain of available commands which are tied to an object
Create a sequence of processing ‘steps’ that are optional and require no particular order
Simpler and cleaner calling code

Sounds good? Let’s get started.

Improving the code in Figure 1

To create method chaining, there are two requirements for our code:

Each method will need to have a return type of the class we’re using
At the end of each method, we’ll need to return the current object instance

Let’s modify the RemoveMayonnaise method with these changes:

     public Fridge RemoveMayonnaise()
     {
         // implementation omitted for brevity
         return this;
     }

Line 1 - Here we’re stating the object type which will be returned
Line 4 - Here we’re returning the instance of itself this from the method

Because we’re returning the object Fridge and we’re returning an instance, we can then access any public method within the Fridge instance, welcome to method chaining. Let’s add the remaining methods for illustration below:

   class Program
    {
        static void Main(string[] args)
        {
            // ### Previous code
            // var fridge = new Fridge();

            // fridge.RemoveMayonnaise();
            // fridge.RemoveCheese();
            // fridge.RemoveHam();
            // fridge.RemoveLettuce();

            // ### New code
            var fridge = new Fridge();

            fridge.RemoveMayonnaise()
                .RemoveHam()
                .RemoveCheese()
                .RemoveLettuce();
        }
    }

    class Fridge
    {
        public Fridge RemoveMayonnaise()
        {
            // implementation omitted for brevity
            return this;
        }

        public Fridge RemoveLettuce()
        {
            // implementation omitted for brevity
            return this;
        }

        public Fridge RemoveCheese()
        {
            // implementation omitted for brevity
            return this;
        }

        public Fridge RemoveHam()
        {
            // implementation omitted for brevity
            return this;
        }
    }

Our calling code under the ### New code comment is smaller, more fluid and looks much cleaner in comparison to ### Previous code.

But Wait…

Jim is back 🧔, the good news is that, thanks to our method chaining, Jim can gather his ingredients in one trip to the fridge.

Now the bad news: from his last sandwich creation, he learned that placing the lettuce at the bottom of the bread makes it moist and threatens the structural integrity of the sandwich, so we need to use the ingredients in a specific order.

But how can we enforce a sequence of events?

Let’s cover that in the next blog post, fluent interfaces!

Micro Multi Tenanted Value Resolver Library 📚

Sun, 07 Jun 2020 00:00:00 +0000

I’ve been recently dealing with enabling multi-tenancy in a full-stack application, the back end being ASP.NET Core. When looking into various multi-tenant libraries I found them to have one or more characteristics:

The APIs were quite complex, and required lots of documentation parsing
Difficult to plug in different value resolutions
Require lots of boilerplate code and implementation of services

Many of these libraries are designed to use configuration files or resolution databases in order to load credentials. I don’t like this, why?

The reason is simple, most credentials don’t belong in files such as appsettings.json. Instead, these should be stored in a Key Vault which has a host of benefits such as permission management, hardware-level encryption and so on. Services such as Azure Key Vault have these benefits.

So, more to the point, I’ve written a very, very small library which implements a Middleware pattern to essentially push specific credentials based upon a tenant resolution pattern into the current HttpContext. The NuGet package can be found here

The nice thing about Azure Key Vault is that all secrets (depending upon access rights) can be loaded into IConfiguration on application build.

More information and the source can be found on GitHub:

https://github.com/BobNudd/MicroMultiTenant

🔃 Angular Network Aware Preload Strategy

Sun, 22 Dec 2019 00:00:00 +0000

Lazy Loading Routes

As your app becomes larger, with different modules to group feature components, a common approach is to lazy load each module in your main routing configuration. Angular recommends this from the start, since it’s a simple yet scalable pattern to load your project’s dependencies once the route is activated. You have likely seen or implemented this pattern in the app-routing.module.ts like below:

import { NgModule } from '@angular/core';
import { RouterModule, Routes } from '@angular/router';

const routes: Routes = [
            {
                path: 'index',
                loadChildren: 'src/app/modules/home/home.module#HomeModule'
            },
            {
                path: 'shop',
                loadChildren: 'src/app/modules/shop/shop.module#ShopModule'
            }
];

import { NgModule } from '@angular/core';
import { RouterModule, Routes } from '@angular/router';

const routes: Routes = [
            {
                path: 'index',
                loadChildren: () => import('src/app/modules/home/home.module').then(m => m.HomeModule)
            },
            {
                path: 'users',
                loadChildren: () => import('src/app/modules/user/user.module').then(m => m.UserModule)
            }
];

Note: the second example is new as of Angular 8, and is a widely supported method of importing modules.

The above examples allow the module to be loaded in when they’re requested via accessing the route tied to the module.

Eager Loading

The standard method of preloading any specified modules as above is to use the PreloadAllModules strategy. This quite simply loads all modules that have been specified in our routes array. The advantage of this method is a better user experience since there isn’t any time lag between an end user clicking a route e.g. /users and the module having to be downloaded.

imports: [
  ... // omitted for brevity
  RouterModule.forRoot(routes,
    { preloadingStrategy: PreloadAllModules })
]

Network Aware Preloading

PreloadAllModules is great on a fast connection, but not everyone is on one. A user on a slow mobile connection, or with data saver switched on, will have every module downloaded in the background whether they ever visit those routes or not, competing with the bandwidth the current page needs.

Angular lets us decide this ourselves by providing a custom PreloadingStrategy. It has a single method, preload, which the router calls for each route that uses loadChildren. Returning load() preloads the module, returning of(null) skips it.

To decide, we can use the Network Information API. navigator.connection exposes effectiveType ('slow-2g', '2g', '3g' or '4g') and saveData, which is true when the user has asked their browser to use less data.

import { Injectable } from '@angular/core';
import { PreloadingStrategy, Route } from '@angular/router';
import { Observable, of } from 'rxjs';

@Injectable({ providedIn: 'root' })
export class NetworkAwarePreloadStrategy implements PreloadingStrategy {
  preload(route: Route, load: () => Observable<any>): Observable<any> {
    return this.hasGoodConnection() ? load() : of(null);
  }

  private hasGoodConnection(): boolean {
    const connection = (navigator as any).connection;

    if (!connection) {
      return true; // API not supported, behave like PreloadAllModules
    }

    if (connection.saveData) {
      return false;
    }

    return !['slow-2g', '2g'].includes(connection.effectiveType);
  }
}

Then swap it in for PreloadAllModules:

imports: [
  ... // omitted for brevity
  RouterModule.forRoot(routes,
    { preloadingStrategy: NetworkAwarePreloadStrategy })
]

Modules we skip aren’t lost, they’re simply loaded on demand when the user visits the route, exactly as in the lazy loading examples above.

Browser Support

At the time of writing, navigator.connection is only available in Chromium based browsers. That’s why the strategy falls back to preloading everything when the API is missing, so those users get the same behaviour as PreloadAllModules. Treat this as a progressive enhancement rather than something to depend on.

Visual Studio Remote Debugger Firewall Error

Thu, 05 Dec 2019 00:00:00 +0000

An Error Occurred while launching remote debugging

This is a relatively common problem (in my experience).

Generally, ensuring that port 4024 for Visual Studio 2019 or 4022 if you’re using 2017 isn’t blocked by your network resolves the issue.

However, in my case, this didn’t work.

Solution

The Web App in question had a custom domain, on the DNS side it was a CNAME, since it was a subdomain.

Finding the IP address of the app (via Properties in Azure) and making two entries in the hosts file on Windows for the CNAME and original *.azurewebsites.net subdomain fixed the issue.

Image Mutation Pipeline - Reshaping and Overlay Images

Mon, 04 Nov 2019 00:00:00 +0000

Overview

In this post, we’ll go over image manipulation by creating a pipeline. The goal of this post is to create an avatar, consisting of a user/character, then surround this picture with a frame as an overlay.

Requirements

Rather than using System.Drawing we’ll instead use ImageSharp which is the most popular image processing library written using .NET Core, it’s very fast and highly optimized. Despite being in beta, I have used it in production for a year without problems.

Using this library, we’ll create our own pipeline of image operations to perform using IImageProcessingContext, and take advantage of some of the heavy lifting ImageSharp provides.

For assets, we’ll be using two images, a base picture and an overlay to surround:

Project Setup

Create a .NET Core Console App
Add the following Package Reference to your .csproj file <PackageReference Include="SixLabors.ImageSharp.Drawing" Version="1.0.0-beta0007" />. If you install through NuGet, ensure that .Drawing is included, else you won’t be able to access the SixLabors.Shapes namespace
Add the assets to your project, make sure you select Copy if newer in the Properties panel > Copy to Output Directory

Code

The first part of this process is to alter the shape of our user image to something that resembles a diamond.

Below is the code for this, with heavy use of comments elaborating on the process since this is much easier than blog annotations:

    static void Main(string[] args)
    {
        // directory to output results
        System.IO.Directory.CreateDirectory("output");

        // wrap operations in using statements to ensure that Dispose is called
        using (var img = Image.Load("me.png"))
        {
            // clone the original image, since we need this for future runs of the application
            using (Image destRound = img.Clone(x => x.AvatarShaping(new Size(225, 225), 150)))
            {
                destRound.Save("output/result.png");
            }
        }
    }

/// <summary>
    /// Here's we're implementing the IImageProcessingContext interface and extending it via an extension method,
    /// This implements a full image mutating pipeline operating on IImageProcessingContext
    /// </summary>
    /// <param name="processingContext">the processingContext</param>
    /// <param name="size">Size of the image</param>
    /// <param name="cornerRadius">corner radius desired, higher will result in a smaller fill</param>
    /// <returns></returns>
    private static IImageProcessingContext AvatarShaping(this IImageProcessingContext processingContext, Size size, float cornerRadius)
    {
        return processingContext.Resize(new ResizeOptions
        {
            Size = size,
            Mode = ResizeMode.Crop
        }).ApplyRoundedCorners(cornerRadius); // the next part of our pipeline
    }

    // This method can be seen as an inline implementation of an `IImageProcessor`:
    // (The combination of `IImageOperations.Apply()` + this could be replaced with an `IImageProcessor`)
    private static IImageProcessingContext ApplyRoundedCorners(this IImageProcessingContext ctx, float cornerRadius)
    {
        Size size = ctx.GetCurrentSize();
        IPathCollection corners = BuildCorners(size.Width, size.Height, cornerRadius);

        var graphicOptions = new GraphicsOptions(true)
        {
           // AlphaCompositionMode = PixelAlphaCompositionMode.DestOut // enforces that any part of this shape that has color is punched out of the background
        };
        // mutating in here as we already have a cloned original
        // use any color (not Transparent), so the corners will be clipped
        return ctx.Fill(graphicOptions, Rgba32.LimeGreen, corners);
    }

    private static IPathCollection BuildCorners(int imageWidth, int imageHeight, float cornerRadius)
    {
        // first create a square
        var rect = new RectangularPolygon(-0.5f, -0.5f, cornerRadius, cornerRadius);

        // then cut out of the square a circle so we are left with a corner
        // Changing the offsets will change how much of the shape is cut out
        IPath cornerTopLeft = rect.Clip(new EllipsePolygon(cornerRadius - 0.5f, cornerRadius - 0.5f, cornerRadius));

        // corner is now a corner shape positions top left
        //lets make 3 more positioned correctly, we can do that by translating the original around the center of the image

        float rightPos = imageWidth - cornerTopLeft.Bounds.Width + 1;
        float bottomPos = imageHeight - cornerTopLeft.Bounds.Height + 1;

        // move it across the width of the image - the width of the shape
        // cut the remaining 3 corners
        IPath cornerTopRight = cornerTopLeft.RotateDegree(90).Translate(rightPos, 0);
        IPath cornerBottomLeft = cornerTopLeft.RotateDegree(-90).Translate(0, bottomPos);
        IPath cornerBottomRight = cornerTopLeft.RotateDegree(180).Translate(rightPos, bottomPos);

        return new PathCollection(cornerTopLeft, cornerBottomLeft, cornerTopRight, cornerBottomRight);
    }

The above code results in the following image:

We can achieve our approximate diamond shape via increasing the cornerRadius float value; however, this will result in less of the canvas showing. Below is a value of 200:

Side note, if you wanted a circle, all that would be required is to set the cornerRadius to 50% of the image size.

Currently we’re filling in the areas we’ve clipped with a solid green. Great for testing, but not for our desired result. You will want to comment out the following line which will automatically remove our colour fill.

AlphaCompositionMode = PixelAlphaCompositionMode.DestOut

Next, we’ll add our overlay, this is simpler than the previous code. We’ll embed another using statement since we still want the original outputs, this makes viewing each stage of the process and making changes much easier whilst we build the pipeline.

Add the following within the destRound using, after destRound.Save("output/fb.png");

using (var overlay = Image.Load("frame.png"))
{
    using (Image overlayClone = destRound.Clone(x => x.GenerateOverlay(overlay)))
    {
        overlayClone.Save("output/complete.png", new PngEncoder()
      {
        ColorType = PngColorType.RgbWithAlpha // needed to instruct the encoder transparency is required
      });
    }
}

GenerateOverlay is part of our pipeline for this project, add the following:

private static IImageProcessingContext GenerateOverlay(this IImageProcessingContext processingContext, Image me)
{
    return processingContext.DrawImage(me, new Point(0, 0), new GraphicsOptions());
}

The above method will draw the image together with the previous image, once that has completed this will be saved to the disk.

The Final Result

It should be noted that the code above is simply a proof of concept, for instance the Clone methods add overhead and shouldn’t appear in production code.

Angular Lazy Loading Modules with Ivy – Uncaught (in promise) error cannot find module

Mon, 28 Oct 2019 00:00:00 +0000

This is a quick post on an error I encountered whilst opting into the Ivy compiler (which should be here Nov 2019).

Once you get through the initial errors, you may see the below in your console log:

core.js:6014 ERROR Error: Uncaught (in promise): Error: Cannot find module 'src/app/modules/module/your-module.module'
Error: Cannot find module 'src/app/modules/module/your-module.module'
    at $_lazy_route_resource lazy namespace object:5
    at ZoneDelegate.invoke (zone-evergreen.js:365)
    at Object.onInvoke (core.js:39707)
    at ZoneDelegate.invoke (zone-evergreen.js:364)
    at Zone.run (zone-evergreen.js:124)
    at zone-evergreen.js:851
    at ZoneDelegate.invokeTask (zone-evergreen.js:400)
    at Object.onInvokeTask (core.js:39688)
    at ZoneDelegate.invokeTask (zone-evergreen.js:399)
    at Zone.runTask (zone-evergreen.js:168)
    at resolvePromise (zone-evergreen.js:793)
    at resolvePromise (zone-evergreen.js:752)
    at zone-evergreen.js:854
    at ZoneDelegate.invokeTask (zone-evergreen.js:400)
    at Object.onInvokeTask (core.js:39688)
    at ZoneDelegate.invokeTask (zone-evergreen.js:399)
    at Zone.runTask (zone-evergreen.js:168)
    at drainMicroTaskQueue (zone-evergreen.js:570)
    at ZoneTask.invokeTask [as invoke] (zone-evergreen.js:485)
    at invokeTask (zone-evergreen.js:1596)

You’re getting this error because Angular 8 introduced a new recommended module loading method; previously the default method of lazy loading modules was to specify a string path to a module:

{path: '/admin', loadChildren: './admin/admin.module#AdminModule'}

This was something built into the Angular toolchain, and performed some wizardry to parse and load the module from the loadChildren attribute.

Since Angular 8, the method of importing modules has changed to the industry recognised / standard, dynamic import. So, your import should now be changed to:

loadChildren: () => import('./admin/admin.module').then(m => m.AdminModule)

Depending upon your IDE choice, you’ll likely get IntelliSense too:

Azure Pipeline- Fixing ‘The project was restored using Microsoft.NETCore.App version x.x.x

Wed, 29 May 2019 00:00:00 +0000

I couldn’t find an answer for this, so I figured a quick post may help others, since people are experiencing this issue.

Initially, despite best practice, explicitly setting a version for the Microsoft.AspNetCore.App package resolved a pipeline library conflict issue, where it could not determine the correct package to restore on the Pipeline side.

Then came the following host of errors:

Error : NETSDK1061: The project was restored using Microsoft.NETCore.App version 1.0.0, but with current settings, version 2.2.0 would be used instead. To resolve this issue, make sure the same settings are used for restore and for subsequent operations such as build or publish. Typically this issue can occur if the RuntimeIdentifier property is set during build or publish but not during restore. For more information, see https://aka.ms/dotnet-runtime-patch-selection.

Some people resolved this issue by specifying their projects to target the latest runtime, however, in my scenario the pipeline apparently still wanted to restore using 1.0.0.

Solution(s)

If you’re using the default generated YAML build script, the generic script the pipeline builds after detecting your project type doesn’t work.

You’ll want to replace

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

with:

- task: DotNetCoreCLI@2
  inputs:
    command: 'restore'
    projects: '**/*.csproj'

Now yes, if you have many projects in your solution this likely would add a good amount of time to your build, but that’s not an issue for me.

Even Easier Solution:

Using the classic editor, you’ll get a visual build task pipeline that should get you 90% of the way there by default. Without changing anything, this almost fully built my default SPA with .NET Core.

Honestly, I feel a little disappointed in the new YAML builder, yes it’s nice to have a version controlled build script, but there are so many people experiencing issues, especially those new to .NET who shouldn’t have to spend hours getting their vanilla .NET Core app to build to a branch or staging site.

Broadcast data strategy with .NET Core, SignalR & Angular

Mon, 06 May 2019 00:00:00 +0000

In today’s post, we’re going to discuss an increasingly common requirement of tasks running within a web application. Depending upon your specialism you can think of these as service workers, or simply background tasks. For this demo we’re going to focus on our web application continuously running a task which emits I/O data for the duration of the app’s lifetime. This data can then be consumed by another service. Since our task needs to run continuously, it’s better suited to an observable pattern on the client side, so we can subscribe and handle new data accordingly. In this scenario, we’re going to use SignalR to trigger invocations on any connected clients.

First, you’ll need to do a small amount of configuration; I won’t elaborate on this since it’s fairly straightforward:

Startup.cs

Add the following to your ConfigureServices(IServiceCollection services)

services.AddSignalR();

Note: In Core 3, you won’t need this call, as the Core Team are tidying up some of the service extensions

Next, you’ll want to add an endpoint for your hub in your Configure(IApplicationBuilder app, IHostingEnvironment env)

app.UseSignalR(routes =>
{
    routes.MapHub<ChangeDataHub>("/changes");
});

With the addition of the ChangeDataHub you’ll need to create a class that inherits from the base class Hub, we’ll leave this blank.

public class ChangeDataHub: Hub
{
}

Good so far? Next we’ll need a Listener service which will manage the HubManager allowing us to send invocation messages to anything which has subscribed to our service.

But, before this we’ll need to think about how we’re going to use our listener service. With .NET Core 1 & 2, Microsoft provided an IWebHost; however, with the addition of 2.1, a new interface, IHost, was introduced. IHost is similar to IWebHost allowing hosted services, DI etc, however it’s more lightweight because it’s designed for background tasks that aren’t used over HTTP. Since we’ll be transmitting our data via web sockets, this is ideal and the right fit for our project.

Registering a host is easy in .NET Core, within our Startup.cs, we’ll want to create a singleton instance of our host. We’ll be implementing IHostedService, which is a great interface that provides just two methods of type Task to implement. Below is the interface:

namespace Microsoft.Extensions.Hosting
{
    //
    // Summary:
    //     Defines methods for objects that are managed by the host.
    public interface IHostedService
    {
        //
        // Summary:
        // Triggered when the application host is ready to start the service.
        Task StartAsync(CancellationToken cancellationToken);
        //
        // Summary:
        // Triggered when the application host is performing a graceful shutdown.
        Task StopAsync(CancellationToken cancellationToken);
    }
}

Plugging in our implementation of the IHostedService as described above:

public void ConfigureServices(IServiceCollection services)
{
      services.AddSingleton<Microsoft.Extensions.Hosting.IHostedService, ChangeDataListenerService>();
}

Now it’s time to implement our interface, like most tasks we don’t want to simply run this once, but on an incremental basis. To achieve this we’ll use the static method Task.Delay which will take our delay value.

private readonly HubLifetimeManager<ChangeDataHub> _hubManager;
private readonly ILogger<DefaultLogger> _logger;
private readonly TimeSpan _delay = TimeSpan.FromSeconds(1);
public ChartListenerService(
    HubLifetimeManager<ChangeDataHub> hubManager,
    ILogger<DefaultLogger> logger)
{
    _hubManager = hubManager;
    _logger = logger;
}
public async Task StartAsync(CancellationToken cts)
{
    _logger.LogInformation("Listener is starting..");
    while (!cts.IsCancellationRequested)
    {
        _logger.LogInformation("Fetching updates..");
        var data = DataManager.Get();
        await _hubManager.SendAllAsync("TransferData", new object[] { data });
        await Task.Delay(_delay, cts);
    }
    _logger.LogInformation("Listener has completed..");
}

That’s our basic implementation completed. It should be noted that if you’re using a repository which for instance uses a DbContext this won’t work, since we’ll be creating a DbContext instance that can become a singleton, which can lead to a whole host of problems.

The solution to this problem is injecting an IServiceScopeFactory instance and using it when we need a scoped instance of a service.

private readonly HubLifetimeManager<ChangeDataHub> _hubManager;
private readonly ILogger<DefaultLogger> _logger;
private readonly TimeSpan _delay = TimeSpan.FromSeconds(1);
private readonly IServiceScopeFactory _scopeFactory;
public ChartListenerService(
    HubLifetimeManager<ChangeDataHub> hubManager,
    ILogger<DefaultLogger> logger,
  IServiceScopeFactory scopeFactory)
{
    _hubManager = hubManager;
    _logger = logger;
  _scopeFactory = scopeFactory;
}
public async Task StartAsync(CancellationToken cts)
{
    _logger.LogInformation("Listener is starting..");
    while (!cts.IsCancellationRequested)
    {
  // Scoped service generation, resolve and dependencies in here
    using (var scope = _scopeFactory.CreateScope())
    {
  
        _logger.LogInformation("Fetching updates..");
    var dataManager = scope.ServiceProvider.GetRequiredService<DataManager>();
    
        await _hubManager.SendAllAsync("TransferData", new object[] { dataManager.Get() });
  }
        await Task.Delay(_delay, cts);
    }
    _logger.LogInformation("Listener has completed..");
}

This also means we don’t have to inject all our service dependencies through the constructor, a good thing in a larger, real-world example.

In the next post, I’ll go over how to connect and consume this information on the client side.

EF Core – Seed Data Strategy

Sun, 06 Jan 2019 00:00:00 +0000

In this post, we’re going to explore the relatively new entity data seeding method which was introduced with EF Core 2.1.

It’s fantastic.

Unlike its predecessor EF6, seed data can be associated with migrations and its relative entity type as part of your migration strategy. What this means is adding or editing any seed data for any entity will compute the correct insert(s) or updates to push those data changes into your DB. This is incredibly powerful and leverages the benefits of tracking that migrations provide.

Let’s see some code

First, you’ll need to override the OnModelCreating method in your DbContext, chances are you’re already doing this in an existing project.

    protected override void OnModelCreating(ModelBuilder builder)
    {
        base.OnModelCreating(builder)   
    }

Next, we’ll call the HasData method and pass in a populated instance of an entity.

    protected override void OnModelCreating(ModelBuilder builder)
    {
        base.OnModelCreating(builder);
        modelBuilder.Entity().HasData(new Person { PersonId = 1, Name = "Ash" });
    }

Good so far? Before running our add-migration command, I want to touch on a few key points:

Primary keys for all entities must be specified; this is how the seed data change detection works, by comparing whether the particular record exists or any fields have changed
Any entities with foreign keys must be explicitly stated
Previously seeded data that has a primary key change will be removed (this makes sense, since the record can no longer be referred to)
You may want custom initialization logic depending on your scenario such as temporary data for testing

Now, running add-migration firstseed will produce a migration for entering our new records (since their primary keys don’t exist in the table).

migrationBuilder.InsertData(
    table: "Persons",
    columns: new[] { "PersonId", "Name" },
    values: new object[] { 1, "Ash" });

Any changes to this data will result in a new migration with correct behaviour (insert/update/delete) making seed data much easier to progress as your application develops, rather than a monolithic script of inserts.

Further Points

Structuring Seed Data

Adding seed data with the OnModelCreating can become difficult to manage, especially with dozens of entities; personally, I would opt to create extension methods to keep this method clean.

    protected override void OnModelCreating(ModelBuilder builder)
    {
        base.OnModelCreating(builder);
        builder.AddContractHoursData();
        builder.AddLocationData();
        builder.AddJobTitleData();
    }

Additionally, creating a single contained class for each data entity was my approach, adhering to the Single Responsibility Principle and violating DRY.

    public static void AddRewardCategoryData(this ModelBuilder model)
    {
        model.Entity().HasData(new List()
        {
            new RewardCategory()
            {
                RewardCategoryId = 1,
                Name = "Vouchers",
                Enabled = true
            },
            new RewardCategory()
            {
                RewardCategoryId = 2,
                Name = "Charity",
                Enabled = true
            }
        });
    }

Creating Identity Users

One question I’ve seen people ask is seeding users from Identity (since UserManager takes care of things like Password Hash).

We can use Identity’s PasswordHasher.

However, there’s a problem with this approach: it will generate a different hash on every run, and therefore bloat every migration with updates. Additionally, you’d rather avoid hard-coding dozens or hundreds of User models.

My strategy for this approach was the following:

Use a library such as Faker to generate a list of user details
Iterate through the list, assign a hard-coded GUID from a list you’ve created
Use the PasswordHasher to generate the hashed password
Pass the completed list to the HasData method
Copy the data within the AspNetUsers table in your database as JSON (I used SQL Operations Studio for this)
Use some Find and replace with regular expressions to fix formatting errors and form into a list of Users
You should end up with a list of populated entities, all with hard-coded data, without tedious data entry

T4 Multiple File Outputs in a .NET Core Project

Sat, 03 Nov 2018 00:00:00 +0000

I recently needed to auto-generate boilerplate classes for a project based upon reflection metadata. It had been a few years since I’d leveraged T4 for SQL Stored Proc generation and the awesome T4MVC template, so surely there’s a shiny new method?

Scripty comes up as an alternative scripting approach for code generation, unfortunately it’s having major issues with VS 2017 and the Roslyn compiler.

So, it looks like T4 Templating is still the way to go. However, whilst writing the template, it appears that splitting code into multiple files and moving directories is quite a problem, as the templating standard behaviour is to nest files under the parent template.

Prerequisites

First, we’re going to leverage the T4 Toolbox extension. Yes, this works in VS 2017 at the time of writing (I have update 15.8.9), you can find this here.

Getting Started

So our strategy here is to have a Runner template and a code generation template.

Runner template – This will be responsible for initialisation of the code generation template, our method is to enumerate through a collection and pass data to our template
Code generation template – This will contain our templating code

Please note – The below represents a basic example, and is not production code.

TemplateRunner.tt

<#@ template language="C#" debug="false" #>          //#A
<#@ output extension="cs" #>                         //#A
<#@ import namespace="System.Linq" #>                //#A
<#@ import namespace="System.Text" #>                //#A
<#@ import namespace="System.IO" #>                  //#A
<#@ import namespace="System.Collections.Generic" #> //#A
<#@ include file="BaseGeneratorApi.tt" #> //#B
<#
BaseGeneratorApi baseTemplate = new BaseGeneratorApi(); //#B
var currentFolder = new FileInfo(Host.ResolvePath(Host.TemplateFile)).DirectoryName; //#C
var webProject = Path.GetFullPath(Path.Combine(currentFolder, @"../", "src")); //#C

var apiControllerNames = File.ReadAllLines(Path.Combine(currentFolder, "bin", "Debug", "netcoreapp2.1", "webapinames.txt")); //#D

if(apiControllerNames != null && apiControllerNames.Any()) {
  
  foreach (string className in apiControllerNames)
  {
    baseTemplate.controllerName = className; //#E
    baseTemplate.controllerNameTitled = char.ToLower(className[0]) + className.Substring(1); //#E

    var controllerFileName = $"{className}Controller.cs";

    baseTemplate.RenderToFile(Path.Combine("Autogenerated", controllerFileName)); #//F
  }
}
#>

Let us go through the code above:

#A – We want to output .cs files, and we’ll need those namespaces to import the required classes for enumeration/file reading
#B – Here we instantiate our code generation template, in this example it’s a template which generates a blank Web API controller
#C – Since the templates base directory at runtime is within a subdirectory of your VS folder, we need to do a little configuration, for this example we want to read a text file in our bin folder
#D – Reading of a text file
#E – We pass a few strings to our own code generation template, since these will be the subject of said class
#F – We call the RenderToFile method, passing a file path for our class

BaseGeneratorApi.tt

<#@ output extension="cs" #>
<#@ include file="T4Toolbox.tt" #> //#A
<#@ assembly name="System.Core" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Text" #>
<#@ import namespace="System.IO" #>
<#@ import namespace="System.Collections.Generic" #>
<#+
public class BaseGeneratorRepository : CSharpTemplate //#B
{
public string repositoryName;
public string repositoryNameTitled;

public override string TransformText() //#B
{
base.TransformText();

#>

//All your code goes here!
[Route("api/[controller]")]
[Authorize]
public class <#= $"{controllerName}Controller" #> : ControllerBase   
{

}

<#+
return this.GenerationEnvironment.ToString();
}
}
#>

As above:

#A – Here we’ll leverage the helpers which T4 Toolbox provides, therefore you’ll need to include the file
#B – We’ll want to inherit from the abstract class CSharpTemplate which is part of the T4 Toolbox. In our derived class we’ll access the TransformText method in the base class to append our additional code

You can have multiple code generation templates which are used within a single template runner, however it increases the scope of a problem occurring in the transformations, and therefore a 1:1 ratio is better.

NopCommerce 3.8 - Wait Operation Timeout

Sun, 17 Jun 2018 00:00:00 +0000

With NopCommerce I’ve noticed high waits on SQL queries resolving when hitting a database in a test environment for local development. This problem becomes more apparent when your seed / test dataset is quite large.

When logging into the Administration area, I would frequently get the following, despite setting a high connection timeout period:

Exception Details: System.ComponentModel.Win32Exception: The wait operation timed out
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[Win32Exception (0x80004005): The wait operation timed out]

The stack trace pointed towards PageList.cs class, the method hit returns an IQueryable DbSet. The initial Administrator dashboard performs a lot of I/O reads which has a large contribution towards the timeout issue.

Solution

All the dashboard statistic calls are within a single View which can be found at:

Nop.Admin/Views/Home/Index.cshtml

To avoid running these on your machine, use the HttpContext.Current.IsDebuggingEnabled and wrap in an If, like below:

<div class="content">
    <div class="row">
        <div class="col-md-12">

            @if (!HttpContext.Current.IsDebuggingEnabled)
            {
                @Html.Widget("admin_dashboard_top")
                if (!Model.IsLoggedInAsVendor)
                {
                    <div class="row">
                        <div class="col-md-12">
                            @Html.Action("NopCommerceNews", "Home")
                        </div>
                    </div>
                }
                @Html.Widget("admin_dashboard_news_after")
                if (!Model.IsLoggedInAsVendor && canManageOrders && canManageCustomers && canManageProducts && canManageReturnRequests)
                {
                    <div class="row">
                        <div class="col-md-12">
                            @Html.Action("CommonStatistics", "Home")
                        </div>
                    </div>
                }
                @Html.Widget("admin_dashboard_commonstatistics_after")
                if (!Model.IsLoggedInAsVendor && (canManageOrders || canManageCustomers))
                {
                    <div class="row">
                        @if (!Model.IsLoggedInAsVendor && canManageOrders)
                        {
                            <div class="col-md-6">
                                @Html.Action("OrderStatistics", "Order")
                            </div>
                        }
                        @if (!Model.IsLoggedInAsVendor && canManageCustomers)
                        {
                            <div class="col-md-6">
                                @Html.Action("CustomerStatistics", "Customer")
                            </div>
                        }
                    </div>
                }
                @Html.Widget("admin_dashboard_customerordercharts_after")
                if (!Model.IsLoggedInAsVendor && canManageOrders)
                {
                    <div class="row">
                        <div class="col-md-8">
                            @Html.Action("OrderAverageReport", "Order")
                        </div>
                        <div class="col-md-4">
                            @Html.Action("OrderIncompleteReport", "Order")
                        </div>
                    </div>
                }
                @Html.Widget("admin_dashboard_orderreports_after")
                if (!Model.IsLoggedInAsVendor && (canManageOrders || canManageProducts))
                {
                    <div class="row">
                        @if (canManageOrders)
                        {
                            <div class="col-md-8">
                                @Html.Action("LatestOrders", "Order")
                            </div>
                        }
                        <div class="col-md-4">
                            @if (canManageProducts)
                            {
                                @Html.Action("PopularSearchTermsReport", "Common")
                            }
                        </div>
                    </div>
                }
                @Html.Widget("admin_dashboard_latestorders_searchterms_after")
                if (canManageOrders)
                {
                    <div class="row">
                        <div class="col-md-6">
                            @Html.Action("BestsellersBriefReportByQuantity", "Order")
                        </div>
                        <div class="col-md-6">
                            @Html.Action("BestsellersBriefReportByAmount", "Order")
                        </div>
                    </div>
                }
                @Html.Widget("admin_dashboard_bottom")
            }

        </div>
    </div>
</div>

Depending on the context and environment of the project, you may want to write a more robust solution, however for local debugging this is perfectly fine.

Fixing MSBuild CS1525, CS1003 Compiler Errors

Fri, 01 Jun 2018 00:00:00 +0000

Recently I needed to use MSBuild to compile a solution for a build server.

The MSBuild was located at:

C:\Program Files (x86)\MSBuild\14.0\Bin

The following errors were occurring despite this working in the past:

error CS1525: Invalid expression term 'decimal'
Syntax error, ',' expected
error CS1003: Syntax error, ',' expected [Web.csproj]

This was due to C# 6/7 features which aren’t supported by this version of MSBuild.

The Fix

Using this version of MSBuild resolved the issue:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin

ChromeDriver for Data Scraping

Mon, 06 Nov 2017 00:00:00 +0000

Selenium

I have used Selenium on occasion for automated frontend tests, it’s my go-to for web automation testing and provides an interface to enable browsers to use the libraries.

ChromeDriver is the implementation of the Chrome browser using the WebDriver protocols developed by Selenium. You can think of this in three blocks:

Chrome – your browser
WebDriver and language protocols – the driver
Executable by the Chromium project – the middleware that you’ll download

Also, it’s worth pointing out that there are implementations for browsers such as Firefox and Edge, handy for cross-browser testing!

But for data scraping?

First, let’s make sure we’re on the same page. By data scraping I mean grabbing information from web pages programmatically, i.e. parsing the HTML markup for that sweet data. Granted, thanks to the popularity of RESTful APIs, we don’t need to do this as much, but it’s still required from time to time.

At an abstract level, you’re making a request for a web page and receiving the markup as a response which can then be parsed; it’s trivial to do in many languages. A basic C# example would be:

using(var wc = new WebClient())
{
    var html = wc.DownloadString("http://somewebsite.com/some-page.html");
}

And use a library such as HtmlAgilityPack for all your parsing and selector needs.

However – Many web apps now use JavaScript-based frameworks such as Angular, React and Vue to pull information into an application client-side. This uses the browser’s JavaScript engine (called V8 in Chrome). Without the JavaScript being executed, we’re just pulling in syntax without content. We’re visiting a house but no one is home.

(It’s worth pointing out that many highly popular websites such as Amazon deliberately obfuscate their markup and rely on JavaScript to parse it, preventing their data from being mined.)

Solving the problem

Here’s where our automated testing tool comes in! As the executable uses Chrome’s libraries, we could use this in theory to render our target website, JavaScript included, and programmatically get the data we want.

Trying this out

For this example I’ll be using the amazon.co.uk website, it should be noted that data scraping is a grey area, and is likely against their ToS. However it serves as a good educational example for otherwise unobtainable data.

Install Chrome if you don’t have it
Download the Chromium exe found here

For this example, I’ll create a simple Console Application

Next we’ll need the Selenium.WebDriver package, which we can find via the NuGet solution window

Using the Selenium library, we’re going to load our Chromium executable and tell Chrome to browse to a product page

Using the FindElementById method, we’ll pass in the id of the element which contains the information we would like. Our finished demo looks like this

The result:

Conclusion

This isn’t a better solution than a code-only approach due to the overhead of creating an instance of Chrome and the loading and rendering of page(s), however it definitely has use cases especially when dealing with heavy dynamic web apps.

ChromeDriver Performance C#

Sun, 05 Nov 2017 00:00:00 +0000

Scenario

So you’ve got a good test suite for automated browser testing, however you’re testing a web app with different user roles, or case scenarios, and that 1 minute test has now grown to 5 minutes and is becoming a bottleneck in the development lifecycle. I ran into this exact problem myself.

In this post, we will be looking at how to reduce elapsed time between operations, and overall reduce total test suite execution time.

The Setup

Web Page

For the post I’ll be using the same website and loading the same web page. I’ve chosen a page with multiple round trips, images, videos, and several megabytes of content to simulate a sizable and real-world web app. Chances are your tests will be fast on static HTML pages regardless of how your tests are written. Below are Google PageSpeed Insights results for this page.

Software

Performance Tests

As we’re making an HTTP request to a remote server, there are many variables that could affect performance outside of our control. Using explicit or implicit waits doesn’t help us here since they’re designed to wait until a specific element appears via polling the DOM. We want to try to eliminate server variance so we’ll use two test sets, one on a local server, and one remote.

Finding UI Elements

Our most common operation will be finding elements on a page and checking if their respective values align with our expectations. The ChromeDriver instance exposes the FindElement and FindElements methods which we’ll be using to find our DOM objects. There are several methods to find elements and we’ll be focusing on the three most popular:

By XPath
By ID
By Class Name

Setup

We’re going to keep this simple and create a blank C# Console Application with the minimum code to test performance, I’ll be using the StopWatch class to store elapsed time for each operation. An arbitrary sample size of 1500 iterations (500 per FindElementBy call) will be used to create an average for our results.

Test Results

Our results demonstrate that finding an element by ID is the most efficient, this is also in line with the documentation.

Optimizations

So our tests so far look good and produce the expected output, we’re going to use the same two tests whilst trying different methods of potential optimization and observe the output.

Browser Extensions

With the amount of HTTP requests ever increasing, especially 3rd party requests, we may benefit from something which filters requests and reduces them. It makes sense that a faster page would mean faster tests. For this test we’re going to use uBlock Origin, available as a Chrome Extension. I’ve chosen uBlock because it has a lightweight footprint and generally makes Chrome leaner via less CPU & RAM consumed by the browser.

Setup

To use the extension we’ll need to grab the .crx file, (you’ll need a third-party tool such as this) and load it into our ChromeDriver instance. Below is the code for this

var cdOptions = new ChromeOptions();
cdOptions.AddExtension(@"C:\temp\uBlock-Origin_v1.14.20.crx");

//Directory where our Chronium executable is located
var cdFilePath = @"C:\tempMonitor";

var cd = new ChromeDriver(cdFilePath, cdOptions, TimeSpan.FromSeconds(30));

Results

We’re going to compare the results with our remote test

The results speak for themselves; we’re loading a remote page and finding elements 200-300 milliseconds quicker due to uBlock. This has another benefit of reducing the probability of a third party HTTP request hanging and causing the page to never finish loading, which will eventually cause a TimeOutException.

Headless Chrome

A new feature of ChromeDriver is the ability to run the browser headless, so you won’t get a GUI, but you’ll still be able to see it in process explorer. Headless has advantages in terms of speed as there’s no rendering of the browser, however naturally there are drawbacks like simulating a user’s input and not being able to see a problem on your page. For brevity we’re going to ignore the pros and cons and simply look at performance impact.

Setup

For this setup we’ll simply be passing in the headless argument, in headless mode the console receives a lot of warnings, so we will hide those for now.

var cdOptions = new ChromeOptions();

cdOptions.AddArgument(@"--headless");
cdOptions.AddArguments(@"--log-level=3");

//Directory where our Chronium executable is located
var cdFilePath = @"C:\tempMonitor";

var cd = new ChromeDriver(cdFilePath, cdOptions, TimeSpan.FromSeconds(30));

Results

With Chrome in headless mode we’ve taken 300-400 milliseconds off our no-extension tests; over a large test suite this would lead to a great performance increase and the ability to run tests in parallel.

Azure FTP 550 - Access is Denied

Sat, 05 Aug 2017 00:00:00 +0000

So, I encountered this for the first time today: a colleague wanted to quickly FTP some files up to an Azure Web App. However, no matter what directory they tried, they always got the same error:

Initially I thought there must be a permission missing from the user account; however, they had got the credentials from a .PublishSettings file for the web app in question.

Solution

Within the XML of the publish settings file, there’s a profile entry for a [insert app name] - Readonly - FTP. This contains credentials for read only access, an easy mistake to make since, depending on your editor, the XML file might be one long line. The correct FTP publish section will just have the word FTP.

Hope that saves someone some time!

HTML5 – Unknown JavaScript Web APIs

Sat, 05 Aug 2017 00:00:00 +0000

There’s a lot of cool things that HTML5 can do, using local storage for data saving/retrieval or acquiring a user’s geolocation for your custom Google map. Well, it turns out there are many, many more JavaScript APIs that I haven’t seen people mention; below are a few of these.

Before we go on, I should mention I’ll be focusing on APIs which benefit mobile/tablet devices, since these are really cool and useful in modern web applications.

Online State

This exposes network connection information to the web; quite simply, we can now determine if a user has an Internet connection within our web application. This is highly useful in cases where apps stream data in, we can let the user know that there’s a connection issue rather than simply returning nothing, creating a much better user experience.

This API is being updated, things like grabbing a connection speed and adjusting the app accordingly would be useful, for example degrading image quality.

window.addEventListener('offline', networkStatus);
window.addEventListener('online', networkStatus);
function networkStatus(e) {
  console.log(e);
}

Support

Browser support for this is excellent for recent browser versions, with some of the smaller device-specific browsers such as Samsung Internet supporting it.

Device Orientation

From this we have an easy way to determine the orientation of a device. This would be especially useful when dealing with media heavy web apps and ensuring your content can use the new orientation real estate efficiently. Also, combined with CSS we can rotate images in a 3D space.

window.addEventListener('deviceorientation', (e) => {
console.log('alpha:', e.alpha);
console.log('beta:', e.beta); 
console.log('gamma:', e.gamma);
});

Support

Browser support for this isn’t great, partial support with known issues. Fine for fun, non-essential features, but I wouldn’t rely on it.

Capture Ambient Light

This is really, really cool. The metric returned from this is lux which is used to measure light. You can read about the sensor API more here. A use case for this would be switching to a darker or lighter theme depending on available light, enabling the user to choose would certainly be a nice feature and provide a better user experience.

window.addEventListener('devicelight', (e) => {
  console.log(`${e.value} lux`);
})

Support

Browser support for this is bad: it has been available in Chrome since version 62, however it is an experimental feature and has to be enabled.

Battery Level

We can now detect battery levels, which has the added benefit of determining if a device is plugged into a power source. The code for this is a little different in that you’re using a promise. Since the W3C example has many code samples, I’ll use one of these.

navigator.getBattery().then(function(battery) {
  console.log(battery.level);
  battery.addEventListener('levelchange', function() {
    console.log(this.level);
  });
});

Support

Browser support for this is pretty poor unless you’re specifically targeting Chrome. Also, as Firefox has removed the feature, it doesn’t look promising for the future.

Writing in BEM using LESS

Sat, 05 Aug 2017 00:00:00 +0000

Overview

This is something I found myself thinking about when using the Block Element Module (BEM) methodology when coding with the LESS preprocessor.

I like this for the following reasons:

Less repeating code
Ability to change the block name and auto cascade to the element descendants
Fewer lines and easier to read

The Code

    block {  
      &__element {
        &--modifier-alpha {
           color: blue;
        }
        &--modifier--beta {
           color: green;
        }
      }
    }

So the compiled elements would look like this:

    block__element--modifier-alpha {
      color: blue;
    }

    block__element--modifier-beta {
      color: green;
    }