RFC v6.0.0: Karton Gateway feature

[psrok1]

Hello,

Karton began as a pretty simple wrapper around redis-py and minio-py to meet the basic needs of building malware analysis microservice pipelines (well... the very first versions actually used RabbitMQ and pika - that was before going open source!). As Karton has grown in features and new performance requirements, the current design has become a huge limitation, making further improvements increasingly difficult.

Right now:

• Fine-grained access control isn’t possible; we’re limited to basic username/password authentication.
• Changing task representation in the backend is cumbersome—HSET would be better so we don’t need to parse the entire JSON just to update a status.
• Scaling beyond the current architecture remains a significant challenge.

To address these issues, version 6.0.0 will introduce a new middleware service: Karton Gateway. It will serve as an interface for all Karton operations. Karton Gateway is a simple FastAPI-based webserver communicating with services via Websockets and REST API. It's designed to be stateless and easily scalable, allowing you to run multiple instances of it behind a load balancer.

Direct Redis/S3 connection will be still supported as a "direct backend", so migration to the new interface can be gradual. However, that way of connecting to Karton will be deprecated and removed in the next major release (v7). v6 version is going to support Karton services 5.0.0+. v7 version will support 6.0.0+ services only.

[msm-cert]

Karton began as a pretty simple wrapper around redis-py and minio-py to

I wanted to correct you, but you already did this in the very next sentence :).

Changing task representation in the backend is cumbersome (...)
To address these issues, version 6.0.0 will introduce (...)

How will karton-gateway help with this? As long as any service connects to redis directly, we are still tied to internal representation tightly. In other words, as long as at least one legacy service (lower right corner) exists, you can't really change anything.

If I understand correctly, you envision a large-scale ecosystem migration, such that every karton service in the future goes only through gateway? What is the timescale and grace period?

mwdb and karton cli connecting to https on the graph

Correct me if i'm wrong, but I'm pretty sure that Karton CLI uses the redis backend == karton backend right now, (not the REST API). Do you plan to rewrite it? Or does it just mean that karton backend will do https operations for the things karton-cli does. How will we implement karton logs using the HTTPs API though? Is there a grace period during which Karton CLI will support both protocols?

v6 version is going to support Karton services 5.0.0+. v7 version will support 6.0.0+ services only.

Do we keep source-level compatibility and migration is a matter of bumping karton-core version, or are there more significant changes required?

More detailed comments in the future, when I actually read the code :).

[psrok1]

If I understand correctly, you envision a large-scale ecosystem migration, such that every karton service in the future goes only through gateway? What is the timescale and grace period?

Yes, exactly. It sounds scary but if we manage to not break anything on the Consumer/Producer source-level (as I checked by making a PoC, it's definitely doable), it should be a matter of starting a new service (gateway) and bump of karton-core dependency in the projects. Dependency bump is still a challenging task, that's why this "grace period" between v6 and v7 releases is pretty important.

Correct me if i'm wrong, but I'm pretty sure that Karton CLI uses the redis backend == karton backend right now, (not the REST API). Do you plan to rewrite it? Or does it just mean that karton backend will do https operations for the things karton-cli does. How will we implement karton logs using the HTTPs API though? Is there a grace period during which Karton CLI will support both protocols?

That's why on the source level we have KartonBackend that implements the operations. If they're not tightly related with Redis-specific things, we don't have to rewrite anything. For example: #300 implements subscribe_logs request in a way that can be almost directly translated to KartonBackend.consume_log call. In my PoC I managed to make a log consumer on top of that.

Do we keep source-level compatibility and migration is a matter of bumping karton-core version, or are there more significant changes required?

We shouid try to keep it source-level compatible as much as possible to not have any obstacles for migration.

I also have some opened question though:

• Right now my PoC was implementing only the Websocket channel. In the same time, we have REST API in https://github.com/CERT-Polska/karton-dashboard. Should we unify karton-dashboard and karton-gateway projects? (I think "yes", but I have not designed it yet)

• I'm a bit worried about usage of S3 presigned URLs. They're really convenient because the download and upload operations can be done via simple HTTP client, so we don't have that botocore mess in all services. But on the other hand, gateway must use the same hostname for S3 as the client (I already made a hack for Docker environments).

  On the other hand, in current protocol design it's just the URL that accepts GET/PUT requests. If we find some huge obstacles with using that, we can just replace these URL with ones pointing at Karton Gateway.

[rakovskij-stanislav]

Hello @psrok1!

This plans for karton-core v6-v7 looks promising! I am curious how can middleware webserver improve the perfomance of the processing? It looks like I'll just DDoS 1-2-5 instances of webservers with my 100-200 workers, and these webservers will DDoS redis -> karton-system instance :)

I fear that by using webservers we will also increase the latency of the work of each task. Another problem will be with big files -- if these web servers are planned to be used also as S3 Resource middleware, we can lose on transfering large files: the balance between RAM consumption, chunk size and overall speed of transfer will affect.

(As I see) The only one thing these web servers resolve is access control which is good idea, but in some seriuos pipelines it is already done by using tls+credentials for minio and redis and nginx+cert for karton-dashboard.

It could be a ok idea to just put more work on Consumers (unpersistant resource clearing, deleting of finished tasks) and make optimizations of karton-system (e.g. don't mark proceed unrouted tasks as Finished - just delete them immediately).

Also we need a formal benchmark. Some small karton-system (karton-playground?) with several dummy services and different strategies of creating tasks (consistent K/s, burst, etc.) like in database benchmarks. This way we can calculate the impact that some features provide.

Thank you for karton-system, this is an incredibly convenient file analysis pipeline!