Python 3 compatibility

[wido]

A lot of Python code inside CloudStack is still Python 2 and we have just 10 months left before Python 2 is no longer supported: https://pythonclock.org/

Various items need to be looked at and checked for Python 3 compatibility. A list, but probably not complete:

• KVM Security Groups
• Xen Security Groups
• Virtual Router scripts (addressed in Update SystemVM scripts from python2 to python3 #3730)
• Various setup tools
• build tools (addressed in some minor changes to make the build run with python 3 #4030)
• marvin framework and smoke tests (Marvin3 #4727)
• marvin component test sets (component test ports/fixes in python3 #5082)
• packaging (cloud.spec) (Suggestions on Python processing of cloud.spec file in packaging directory #4513 is asking for it)

These are isolated projects and can usually be taken care of separately and also properly tested.

Creating this ticket so it can be referred to and also this is out there.

[yadvr]

Is there an argument to say, these may be implemented purely in Java (if possible)? Currently, many of these scripts esp on KVM are forked via the cloudstack-agent and called.
For VR, I played with GRPC and I think it can be secured against the current CA-framework introduced in 4.11; here's a PoC: shapeblue#44
I'll propose more details on dev@ once I can put together design/idea details.

[falcon78921]

@rhtyd @wido I would love to help with moving over to Python 3, if that's still the plan.

[wido]

Although I agree with @rhtyd that a lot can be done in Java, it's not always the easiest. Python is sometimes a lot easier to develop and talk directly with the system for executing commands.

We haven't started on this @falcon78921 , but any help is welcome! Just pick a Python script and start working on it.

[yadvr]

@wido changing the env dependency is a big step, python3.x is not available on all distros by default. I think it's better to discuss on dev@ for future steps. In my proposal, for the VR for example I've proposed that we move to a lightweight agent and as PoC I've shared a Go-Java approach based on grpc.

I think the best time would be to discuss/start doing things after 4.13, as we've decided to deprecate CentOS6 after 4.13.

[wido]

@rhtyd Python 3 is available on ALL distros. It has been for a very long time. It's on RHEL, CentOS, Debian and Ubuntu. No need to install anything :)

We can always talk about this, but the fact is that per January Python 2 is EOL and it's not so super difficult to write things in Python 3.

I'm not against an Agent in the VR for example, sounds like a good thing!

[PaulAngus]

There are also knock-on effects, ie I suspect that the chances of running Python3 on XenServer 6.x is near zero.

[wido]

@PaulAngus Python 2 is EOL in 2020, so any plans on what Xen is going to do?

But this doesn't stop us to fix other Python code before that time.

[yadvr]

I think we'll need to deprecate all the XS 6.x, which we can discuss and aim to do in 4.14 and onwards. XS7 or xcp-ng 7.x with CentOS7 based dom0 may be more easy to support python3.