Amethyst Platform - DevLog 2

Posted on 2021-05-22 byVlad Călin

Reading time of 1 minutes

Amethyst Platform is one of my personal attempts at launching a product. Every couple of weeks, I will post an update with the progress so far. At this stage, I am still working towards building and launching an MVP.

What is Amethyst Platform?

Amethyst Platform strives to be a one-stop platform for the complete application lifecycle. As soon as you push some changes to your code, we test, build, configure and deploy it so that you don't have to worry about that. Everything is managed through an amethyst.yaml file in your project, and we do the rest. You can read more on the Amethyst Platform website or on the Amethyst Platform project page.

Progress in the past two weeks

Unfortunately, in the past two weeks, I didn't manage to work as much as I wanted on Amethyst Platform due to the amount of work involved with my current job, Vuuh.com. I had to invest a big chunk of my time and energy in pushing our product forward by squishing a lot of last-minute bugs, and some new high impact features.

By doing that, I didn't have enough energy left to make significant progress with Amethyst Platform in my spare time, but nonetheless, some progress has been done.

Job logs using ElasticSearch

I completed the migration of the system that handles the logs of the pipeline jobs, by pushing them in real time to an ElasticSearch cluster, with the purpose of being able to search in them, store them for longer, and most importantly, retrieve them faster.

Indeed, by doing this change, the usual load time of the logs dropped from 2-3 seconds (the time needed for the Kubernetes log retrieval API call through which all logs are retrieved at once) to ~100-200ms.

I also implemented an extension of this logging mechanism, so that I can emit system events when I need to, in order to be able to see things that happen outside the job pods, such as Kubernetes resource creation.

Admin interface

Over the many and long debugging sessions, I realized that I spend too much time doing manual kubectl commands to check out if the cluster resources created properly. This involved a lot of guesswork from my side, and long kubectl commands with various filters, things that overall took too much time.

Sooner or later, I would have needed an admin interface, where superusers of the platform could inspect what is happening under the hood. The Django admin wasn't enough for this job because the majority of the debugging time was spent inspecting Kubernetes resources rather than Django model instances.

So I started the groundwork for building a specialized admin interface for Amethyst Platform, so that I would be able to navigate faster inside the Kubernetes cluster, see what resources are attached to various business logic entities, and intervene when needed. Basically, it will become the de-facto platform used for technical support.

Right now, only the read-only functionality has been implemented, following to implement the rest during the next two weeks.

Next steps

Although I would need to push harder the next two weeks to make up for the slow progress of these past two weeks, I foresee some pretty hectic days ahead again with my current job.

To add more on top of that, I also enrolled in a game jam (The GoGodotJam), to experiment a little with the idea of game making. I have a lot of started game prototypes, but I never had the discipline to build something more polished and release it to the public, mostly because I chose a game scope way too large for a single person. But with this game jam, I look forward to making something smaller, following a set theme, with the purpose of just building something small and release it (no more over ambitious projects).

On the Amethyst Platform front, I'll try to finish the basic admin interface features to unlock a better in-cluster debugging experience, and enforce workload resource limits (global limits for pipeline jobs, configurable limits for deployed services). I also plan to write one blog article for the Amethyst Platform Blog, to get that content marketing game going.